2024-03-29T11:08:36.904 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:08:36.904 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/orig.config.yaml 2024-03-29T11:08:36.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:08:37.022 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622696' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGpptLcC57lUuS0F5ZaBYanQ/mD3T/64LfMnxx3oNg5dHdHQtLjUnvP8USSCCr61/p4NtkvfPKrD8ypNlloLkw= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAxy1Of+RLhQGS1byxOnbjm9Sof/8KrE4Eb13V1+djlQp3zjp6NipL3pg8z+KxDbrJI5ZuZbWw2almQYuUn85mc= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:08:37.022 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:08:37.022 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:08:37.023 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:08:37.024 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:08:37.047 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:08:37.047 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:08:37.047 INFO:teuthology.packaging:ref: None 2024-03-29T11:08:37.048 INFO:teuthology.packaging:tag: None 2024-03-29T11:08:37.048 INFO:teuthology.packaging:branch: main 2024-03-29T11:08:37.048 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:08:37.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:08:37.296 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:08:37.298 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:08:37.304 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:08:37.304 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:08:37.310 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:08:37.323 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:08:37.329 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:08:37.355 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.977449', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGpptLcC57lUuS0F5ZaBYanQ/mD3T/64LfMnxx3oNg5dHdHQtLjUnvP8USSCCr61/p4NtkvfPKrD8ypNlloLkw='} 2024-03-29T11:08:37.376 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.978867', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAxy1Of+RLhQGS1byxOnbjm9Sof/8KrE4Eb13V1+djlQp3zjp6NipL3pg8z+KxDbrJI5ZuZbWw2almQYuUn85mc='} 2024-03-29T11:08:37.377 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:08:37.383 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:08:37.383 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:08:37.383 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:08:37.465 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0b0af860d0>, signals=[15]) 2024-03-29T11:08:37.465 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:08:37.472 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:08:37.472 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:08:37.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:37.564 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:08:37.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:37.661 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:08:37.668 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-03-29T11:08:37.686 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-03-29T11:08:37.686 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-03-29T11:08:37.742 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="9" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="9" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:08:37.743 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:08:37.744 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-03-29T11:08:37.765 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-03-29T11:08:37.783 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-03-29T11:08:37.783 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-03-29T11:08:37.840 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:08:37.841 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:08:37.841 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:08:37.841 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:08:37.841 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:08:37.841 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:08:37.841 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-03-29T11:08:37.859 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:08:37.868 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:08:37.874 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:08:37.874 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:08:37.876 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:08:37.897 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:08:37.903 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:08:37.903 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:08:37.932 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:08:38.050 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:08:38.106 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:08:38.123 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:08:38.123 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:08:38.123 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:08:38.124 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:08:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:08:38.141 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-03-29T11:08:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:08:38.142 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-29T11:08:38.197 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:08:38.198 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.14.0-361.el9.x86_64 2024-03-29T11:08:38.198 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum install -y kernel 2024-03-29T11:08:38.199 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:08:38.199 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-361.el9.x86_64 2024-03-29T11:08:38.199 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-03-29T11:08:40.236 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:08:40.454 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:08:41.662 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-29T11:08:42.079 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T11:08:42.427 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-29T11:08:43.581 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 7.2 MB/s | 8.0 MB 00:01 2024-03-29T11:08:45.114 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T11:08:49.379 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.1 MB 00:00 2024-03-29T11:08:50.914 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-29T11:08:53.226 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 21 MB 00:02 2024-03-29T11:08:55.520 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 1.7 MB/s | 19 MB 00:10 2024-03-29T11:08:58.190 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T11:08:59.420 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-29T11:09:00.480 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:09:00.480 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:09:00.586 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:09:00.597 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:09:00.597 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:09:00.597 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:09:00.597 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:09:00.597 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:09:00.597 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:09:00.597 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:09:00.598 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:09:00.598 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:09:00.598 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:09:00.598 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:09:00.598 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:09:00.598 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-03-29T11:09:00.598 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:09:00.599 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 76 M 2024-03-29T11:09:00.599 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 123 M 2024-03-29T11:09:00.599 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:09:00.741 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:09:00.904 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-29T11:09:01.242 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T11:09:01.817 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T11:09:02.225 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:09:02.229 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:02.229 INFO:teuthology.orchestra.run.smithi152.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-29T11:09:02.229 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:09:02.492 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:09:02.605 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:09:02.606 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:09:03.449 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:09:03.450 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:09:05.794 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:09:06.480 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:06.572 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:07.473 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T11:09:09.349 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:09.493 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:09.830 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:09:09.830 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:09:09.925 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:09:09.935 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:09:09.936 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:09:09.936 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:09:09.936 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:09:09.936 INFO:teuthology.orchestra.run.smithi138.stdout:Install 4 Packages 2024-03-29T11:09:09.936 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:09:09.936 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 76 M 2024-03-29T11:09:09.937 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 123 M 2024-03-29T11:09:09.937 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:09:10.099 INFO:teuthology.orchestra.run.smithi138.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:09:10.541 INFO:teuthology.orchestra.run.smithi138.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 25 MB/s | 15 MB 00:00 2024-03-29T11:09:11.141 INFO:teuthology.orchestra.run.smithi138.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T11:09:11.558 INFO:teuthology.orchestra.run.smithi138.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:09:11.561 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:11.561 INFO:teuthology.orchestra.run.smithi138.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-29T11:09:11.562 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:09:11.936 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:09:11.936 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:09:12.749 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:09:12.750 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:09:14.235 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:14.267 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:09:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:15.745 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:18.466 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:18.614 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:18.970 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:23.447 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:28.159 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:48.316 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:48.316 INFO:teuthology.orchestra.run.smithi152.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:48.316 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:09:48.360 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:48.778 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:48.778 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:48.778 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:48.778 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:49.204 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:49.204 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:09:49.204 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:09:49.204 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:49.204 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:49.204 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:49.204 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:49.205 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:09:49.205 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:09:49.518 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:49.611 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:49.611 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:49.612 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:49.612 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:49.612 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:49.612 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:09:49.612 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:49.612 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:49.693 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:49.693 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:49.693 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-03-29T11:09:49.727 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-03-29T11:09:49.727 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-03-29T11:09:49.727 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-03-29T11:09:49.727 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 68.el9 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8264254 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:09:49.728 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:49.729 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:49.729 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:49.729 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-03-29T11:09:49.729 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:09:49.729 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:49.729 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:49.730 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:49.730 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:50.323 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-03-29T11:09:51.805 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:51.836 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-03-29T11:09:51.839 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-03-29T11:09:51.854 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.ynDDlfuyn3 2024-03-29T11:09:51.854 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ynDDlfuyn3 2024-03-29T11:09:51.922 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.ynDDlfuyn3 2024-03-29T11:09:52.056 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.ynDDlfuyn3 is 6KB 2024-03-29T11:09:52.079 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.ynDDlfuyn3 2024-03-29T11:09:52.093 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:52.157 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:52.157 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:52.157 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:52.158 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:52.159 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:52.294 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-03-29T11:09:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:57.584 INFO:teuthology.orchestra.run.smithi138.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:09:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:57.990 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:57.991 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:57.991 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:57.991 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:58.442 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:58.442 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:09:58.442 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:09:58.442 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:58.443 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:58.443 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:58.443 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:58.443 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:09:58.443 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:09:58.782 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:58.883 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:58.884 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:58.884 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:58.884 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:58.884 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:58.885 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:09:58.885 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:58.885 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:58.966 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:58.966 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:58.966 DEBUG:teuthology.orchestra.run.smithi138:> sudo rpm -qi grub2-tools 2024-03-29T11:09:59.004 INFO:teuthology.orchestra.run.smithi138.stdout:Name : grub2-tools 2024-03-29T11:09:59.004 INFO:teuthology.orchestra.run.smithi138.stdout:Epoch : 1 2024-03-29T11:09:59.004 INFO:teuthology.orchestra.run.smithi138.stdout:Version : 2.06 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Release : 68.el9 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: x86_64 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Group : Unspecified 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Size : 8264254 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:License : GPLv3+ 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Packager : builder@centos.org 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Vendor : CentOS 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:59.005 INFO:teuthology.orchestra.run.smithi138.stdout:Description : 2024-03-29T11:09:59.006 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:09:59.006 INFO:teuthology.orchestra.run.smithi138.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:59.006 INFO:teuthology.orchestra.run.smithi138.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:59.006 INFO:teuthology.orchestra.run.smithi138.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:59.006 INFO:teuthology.orchestra.run.smithi138.stdout:hardware devices. 2024-03-29T11:09:59.006 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:09:59.006 INFO:teuthology.orchestra.run.smithi138.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:59.007 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:59.007 INFO:teuthology.task.kernel:Updating grub on smithi138 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:59.007 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:59.576 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2024-03-29T11:10:00.942 INFO:teuthology.orchestra.run.smithi138.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:10:00.982 INFO:teuthology.orchestra.run.smithi138.stderr:done 2024-03-29T11:10:00.984 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-03-29T11:10:00.999 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.X8gSOtUjCY 2024-03-29T11:10:00.999 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.X8gSOtUjCY 2024-03-29T11:10:01.081 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.X8gSOtUjCY 2024-03-29T11:10:01.221 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.X8gSOtUjCY is 6KB 2024-03-29T11:10:01.268 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.X8gSOtUjCY 2024-03-29T11:10:01.282 DEBUG:teuthology.orchestra.run.smithi138:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:10:01.347 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:10:01.347 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:10:01.347 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:10:01.348 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:10:01.348 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:10:01.522 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2024-03-29T11:10:01.622 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-29T11:10:22.318 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:10:22.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-29T11:10:22.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:31.534 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:10:31.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-29T11:10:31.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:40.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-03-29T11:10:49.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-29T11:10:49.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:49.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-29T11:10:52.738 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-03-29T11:10:58.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-29T11:10:58.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:01.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-29T11:11:04.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-29T11:11:04.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:13.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-29T11:11:13.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:23.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-03-29T11:11:38.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-29T11:11:38.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:41.408 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-29T11:11:41.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-29T11:11:41.877 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:41.877 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-29T11:11:41.891 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:41.891 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:41.892 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:41.892 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:42.893 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:11:42.893 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:42.983 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-03-29T11:11:43.025 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:45.865 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-03-29T11:11:46.233 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-29T11:11:46.233 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:46.233 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-03-29T11:11:46.247 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:46.248 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:46.248 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:46.248 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:47.248 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:11:47.249 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:47.341 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2024-03-29T11:11:47.403 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:47.403 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:11:47.413 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:11:47.414 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:47.417 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:47.439 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:11:47.450 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:11:47.464 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:11:47.464 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:47.482 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:47.537 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:11:47.552 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:11:47.553 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:47.555 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:47.603 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:47.623 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:47.705 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:47.718 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:47.720 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:11:47.728 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:11:47.729 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:47.731 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:47.773 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:11:47.787 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:11:47.788 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:47.790 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:47.817 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:47.909 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:47.979 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:48.034 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:48.095 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:11:48.095 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:48.128 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:48.183 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:48.260 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:48.313 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:48.387 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:11:48.387 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:48.432 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-03-29T11:11:48.434 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-03-29T11:11:48.552 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:48.657 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:49.043 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:11:49.051 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:11:49.051 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:11:49.068 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:11:49.084 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2024-03-29T11:11:49.154 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2024-03-29T11:11:49.154 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-03-29T11:11:49.499 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-03-29T11:11:49.501 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:11:49.501 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2024-03-29T11:11:49.530 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2024-03-29T11:11:49.531 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-03-29T11:11:49.613 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-03-29T11:11:49.613 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi138.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:11:49.614 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:49.645 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:49.676 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:11:49.676 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:49.722 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:49.774 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:11:49.784 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:11:49.801 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:11:49.802 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi138.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:16:15.035 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi138.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-03-29T11:16:15.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-29T11:16:15.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:15.125 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-03-29T11:16:15.208 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-29T11:16:15.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-29T11:16:15.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:15.296 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-29T11:16:15.377 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-29T11:16:15.378 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:16:15.388 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:16:15.389 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:16:15.389 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:16:15.391 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:16:15.392 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:16:15.421 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:16:15.438 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:16:15.449 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:16:15.465 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:16:15.467 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2024-03-29T11:16:15.481 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2024-03-29T11:16:15.495 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-03-29T11:16:15.497 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:16:15.508 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-03-29T11:16:15.514 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:16:15.524 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:16:15.540 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:16:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:16:15.569 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:16:15.570 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-03-29T11:16:15.570 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.570 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.570 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.570 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.595 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:16:15.599 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:16:15.599 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-29T11:16:15.599 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.599 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.599 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.599 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:15.600 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:16:15.608 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:16:15.608 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:16:15.608 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:16:15.608 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:16:15.613 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:16:15.613 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:16:15.613 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:16:15.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:16:15.616 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:16:15.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:16:15.956 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:16:15.956 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:16:15.957 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:16:15.958 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:16:16.070 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:16:16.071 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:16:16.071 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:16:16.072 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:16:16.073 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:16:16.073 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:16:16.100 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:16:16.101 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:16:16.102 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:16:16.102 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:16:16.169 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:16:16.170 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:16:16.249 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:16:16.250 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:16:16.316 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-03-29T11:16:16.317 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-03-29T11:16:16.318 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2024-03-29T11:16:16.319 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2024-03-29T11:16:16.569 INFO:teuthology.orchestra.run.smithi138.stdout:74 files removed 2024-03-29T11:16:16.571 INFO:teuthology.orchestra.run.smithi152.stdout:74 files removed 2024-03-29T11:16:16.596 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw 2024-03-29T11:16:16.601 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-03-29T11:16:17.566 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 122 kB/s | 82 kB 00:00 2024-03-29T11:16:17.663 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 108 kB/s | 82 kB 00:00 2024-03-29T11:16:18.200 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-29T11:16:18.397 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 15 kB/s | 10 kB 00:00 2024-03-29T11:16:18.900 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-03-29T11:16:18.939 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-29T11:16:19.503 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:16:19.559 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:16:21.003 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-29T11:16:21.051 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-29T11:16:22.069 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:16:22.102 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T11:16:24.101 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-29T11:16:24.131 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-29T11:16:28.202 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-29T11:16:28.354 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.1 MB 00:00 2024-03-29T11:16:29.688 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 16 kB 00:00 2024-03-29T11:16:29.761 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-29T11:16:31.007 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:16:31.154 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T11:16:36.006 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:16:36.129 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T11:16:38.628 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:16:38.646 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:16:38.646 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:16:38.646 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:16:38.646 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:16:38.646 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:38.647 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:38.648 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout:Installing weak dependencies: 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout:Install 30 Packages 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 2 Packages 2024-03-29T11:16:38.649 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:16:38.650 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 58 M 2024-03-29T11:16:38.650 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:16:38.743 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:16:38.756 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:38.757 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:38.758 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout:Install 30 Packages 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-03-29T11:16:38.759 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:16:38.760 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 58 M 2024-03-29T11:16:38.760 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:16:42.932 INFO:teuthology.orchestra.run.smithi138.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 5.3 MB 00:04 2024-03-29T11:16:43.040 INFO:teuthology.orchestra.run.smithi138.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 233 kB/s | 25 kB 00:00 2024-03-29T11:16:43.757 INFO:teuthology.orchestra.run.smithi138.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 742 kB 00:00 2024-03-29T11:16:44.232 INFO:teuthology.orchestra.run.smithi138.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.0 MB/s | 498 kB 00:00 2024-03-29T11:16:48.237 INFO:teuthology.orchestra.run.smithi152.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 9.7 MB 00:09 2024-03-29T11:16:48.354 INFO:teuthology.orchestra.run.smithi152.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-29T11:16:49.179 INFO:teuthology.orchestra.run.smithi152.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 899 kB/s | 742 kB 00:00 2024-03-29T11:16:49.763 INFO:teuthology.orchestra.run.smithi152.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 855 kB/s | 498 kB 00:00 2024-03-29T11:16:50.698 INFO:teuthology.orchestra.run.smithi138.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 808 kB/s | 5.1 MB 00:06 2024-03-29T11:16:50.890 INFO:teuthology.orchestra.run.smithi138.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 236 kB/s | 45 kB 00:00 2024-03-29T11:16:51.105 INFO:teuthology.orchestra.run.smithi152.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 449 kB/s | 5.3 MB 00:12 2024-03-29T11:16:51.141 INFO:teuthology.orchestra.run.smithi138.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 525 kB/s | 131 kB 00:00 2024-03-29T11:16:51.205 INFO:teuthology.orchestra.run.smithi152.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-29T11:16:51.382 INFO:teuthology.orchestra.run.smithi138.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 670 kB/s | 162 kB 00:00 2024-03-29T11:16:51.455 INFO:teuthology.orchestra.run.smithi152.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 525 kB/s | 131 kB 00:00 2024-03-29T11:16:51.574 INFO:teuthology.orchestra.run.smithi138.stdout:(9/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 775 kB/s | 9.7 MB 00:12 2024-03-29T11:16:51.789 INFO:teuthology.orchestra.run.smithi152.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 486 kB/s | 162 kB 00:00 2024-03-29T11:16:51.900 INFO:teuthology.orchestra.run.smithi138.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 620 kB/s | 320 kB 00:00 2024-03-29T11:16:52.133 INFO:teuthology.orchestra.run.smithi138.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 533 kB/s | 298 kB 00:00 2024-03-29T11:16:52.192 INFO:teuthology.orchestra.run.smithi138.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 340 kB/s | 99 kB 00:00 2024-03-29T11:16:52.251 INFO:teuthology.orchestra.run.smithi138.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 907 kB/s | 106 kB 00:00 2024-03-29T11:16:52.276 INFO:teuthology.orchestra.run.smithi138.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T11:16:52.301 INFO:teuthology.orchestra.run.smithi138.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-29T11:16:52.322 INFO:teuthology.orchestra.run.smithi152.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 601 kB/s | 320 kB 00:00 2024-03-29T11:16:52.352 INFO:teuthology.orchestra.run.smithi138.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T11:16:52.377 INFO:teuthology.orchestra.run.smithi138.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:16:52.402 INFO:teuthology.orchestra.run.smithi138.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 832 kB/s | 41 kB 00:00 2024-03-29T11:16:52.427 INFO:teuthology.orchestra.run.smithi138.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:16:52.444 INFO:teuthology.orchestra.run.smithi138.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 836 kB/s | 35 kB 00:00 2024-03-29T11:16:52.461 INFO:teuthology.orchestra.run.smithi138.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:16:52.823 INFO:teuthology.orchestra.run.smithi152.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 596 kB/s | 298 kB 00:00 2024-03-29T11:16:53.015 INFO:teuthology.orchestra.run.smithi152.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 518 kB/s | 99 kB 00:00 2024-03-29T11:16:53.124 INFO:teuthology.orchestra.run.smithi152.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-29T11:16:53.174 INFO:teuthology.orchestra.run.smithi152.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T11:16:53.207 INFO:teuthology.orchestra.run.smithi152.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:16:53.266 INFO:teuthology.orchestra.run.smithi152.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:16:53.308 INFO:teuthology.orchestra.run.smithi152.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-29T11:16:53.333 INFO:teuthology.orchestra.run.smithi152.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-29T11:16:53.376 INFO:teuthology.orchestra.run.smithi152.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:16:53.401 INFO:teuthology.orchestra.run.smithi152.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T11:16:53.426 INFO:teuthology.orchestra.run.smithi152.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-29T11:16:53.851 INFO:teuthology.orchestra.run.smithi152.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 727 kB/s | 309 kB 00:00 2024-03-29T11:16:54.193 INFO:teuthology.orchestra.run.smithi152.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-29T11:16:54.269 INFO:teuthology.orchestra.run.smithi152.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-29T11:16:54.344 INFO:teuthology.orchestra.run.smithi152.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-29T11:16:54.419 INFO:teuthology.orchestra.run.smithi152.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 899 kB/s | 67 kB 00:00 2024-03-29T11:16:54.503 INFO:teuthology.orchestra.run.smithi152.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-29T11:16:54.578 INFO:teuthology.orchestra.run.smithi152.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-29T11:16:54.678 INFO:teuthology.orchestra.run.smithi152.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-29T11:16:56.121 INFO:teuthology.orchestra.run.smithi152.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 822 kB/s | 5.1 MB 00:06 2024-03-29T11:16:56.194 INFO:teuthology.orchestra.run.smithi138.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 4.4 MB 00:03 2024-03-29T11:16:56.245 INFO:teuthology.orchestra.run.smithi138.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-03-29T11:16:56.295 INFO:teuthology.orchestra.run.smithi138.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-03-29T11:16:56.362 INFO:teuthology.orchestra.run.smithi138.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:16:56.462 INFO:teuthology.orchestra.run.smithi138.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-29T11:16:56.529 INFO:teuthology.orchestra.run.smithi138.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T11:16:56.671 INFO:teuthology.orchestra.run.smithi138.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-29T11:16:59.612 INFO:teuthology.orchestra.run.smithi152.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 700 kB/s | 3.4 MB 00:04 2024-03-29T11:16:59.946 INFO:teuthology.orchestra.run.smithi138.stdout:(29/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 41 kB/s | 309 kB 00:07 2024-03-29T11:16:59.954 INFO:teuthology.orchestra.run.smithi152.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 836 kB/s | 3.1 MB 00:03 2024-03-29T11:17:01.454 INFO:teuthology.orchestra.run.smithi138.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 722 kB/s | 3.4 MB 00:04 2024-03-29T11:17:04.571 INFO:teuthology.orchestra.run.smithi138.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 693 kB/s | 3.1 MB 00:04 2024-03-29T11:17:06.204 INFO:teuthology.orchestra.run.smithi152.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 746 kB/s | 20 MB 00:27 2024-03-29T11:17:06.208 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:06.208 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.1 MB/s | 58 MB 00:27 2024-03-29T11:17:06.209 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:17:06.248 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:17:06.248 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:17:06.721 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:17:06.721 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:17:08.198 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:17:08.229 INFO:teuthology.orchestra.run.smithi138.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 690 kB/s | 20 MB 00:29 2024-03-29T11:17:08.232 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:08.233 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.0 MB/s | 58 MB 00:29 2024-03-29T11:17:08.233 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:17:08.271 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:17:08.271 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:17:08.285 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:17:08.317 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:17:08.342 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:17:08.365 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:08.382 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:08.567 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:08.622 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:08.738 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:17:08.738 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:17:08.964 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:17:09.038 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:09.136 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:09.234 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:17:09.262 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:09.325 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:09.354 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:17:09.398 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:17:09.467 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:09.498 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:09.545 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:09.580 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:17:09.693 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:17:09.731 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:17:09.772 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:17:09.805 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:17:09.834 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:17:09.866 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:17:10.178 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:17:10.205 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:10.211 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:17:10.278 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:17:10.309 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:17:10.317 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:17:10.342 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:17:10.354 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:17:10.365 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:10.382 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:10.808 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:17:10.832 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:10.904 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:10.960 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:17:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:17:11.010 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:17:11.035 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:11.242 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:17:11.270 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:11.333 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:11.362 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:17:11.406 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:17:11.475 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:11.497 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:11.561 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:11.595 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:17:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:17:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:17:11.788 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:17:11.820 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:17:11.849 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:17:11.882 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:17:12.197 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:17:12.228 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:17:12.387 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:17:12.840 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:17:12.872 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:12.943 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:13.011 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:17:13.041 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:17:13.152 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:13.190 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:13.734 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:13.769 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:13.862 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:13.862 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:17:13.862 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:17:13.862 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:17:13.904 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:15.232 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:15.279 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:15.830 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:15.874 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:15.967 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:17:15.967 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:17:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:17:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-03-29T11:17:23.132 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:17:24.037 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:24.061 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:24.061 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:24.061 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:17:24.061 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:24.061 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:24.061 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:17:24.063 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:24.148 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:24.148 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:25.288 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:17:25.289 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:25.291 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:17:25.292 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /root 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2024-03-29T11:17:25.590 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:17:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:17:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:17:26.378 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:17:26.478 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:26.501 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:26.501 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:26.501 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:17:26.502 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:26.502 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:26.502 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:17:26.503 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:26.736 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-03-29T11:17:27.292 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:51 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:17:27.441 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:17:27.442 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:17:27.442 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:17:27.442 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:17:27.445 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:17:27.447 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 48 M 2024-03-29T11:17:27.447 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 204 M 2024-03-29T11:17:27.447 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:17:27.745 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:17:27.746 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:17:27.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:17:27.962 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-29T11:17:28.004 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T11:17:28.071 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-29T11:17:28.530 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 199 kB/s | 163 kB 00:00 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.964 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:17:28.965 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:17:28.966 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:17:28.967 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:17:29.294 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-test 2024-03-29T11:17:29.855 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:17:30.008 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:17:30.011 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 48 M 2024-03-29T11:17:30.011 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 204 M 2024-03-29T11:17:30.011 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:17:30.222 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:17:30.273 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-29T11:17:30.306 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-29T11:17:30.548 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-03-29T11:17:30.724 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 249 kB/s | 163 kB 00:00 2024-03-29T11:17:52.705 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 47 MB 00:22 2024-03-29T11:17:52.708 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:52.708 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.1 MB/s | 48 MB 00:22 2024-03-29T11:17:52.723 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:17:52.744 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:17:52.744 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:17:52.960 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 47 MB 00:25 2024-03-29T11:17:52.963 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:52.963 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.9 MB/s | 48 MB 00:25 2024-03-29T11:17:52.976 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:17:52.996 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:17:52.996 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:17:53.119 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:17:53.119 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:17:53.388 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:17:53.389 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:17:53.697 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:17:53.764 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:53.814 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:53.856 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:53.905 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:53.934 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:53.982 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:17:54.069 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:54.112 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:54.152 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:54.193 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:54.221 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:58.467 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:58.544 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:58.760 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:18:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:18:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:18:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:18:01.075 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:01.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:18:01.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:18:01.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:01.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:18:01.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:01.330 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:01.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:18:01.487 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:01.487 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:01.488 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:01.648 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph 2024-03-29T11:18:01.798 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-03-29T11:18:02.216 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:02.369 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:02.372 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-29T11:18:02.372 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:02.372 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-29T11:18:02.372 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:02.372 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:18:02.372 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:18:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-29T11:18:02.375 INFO:teuthology.orchestra.run.smithi138.stdout:Install 36 Packages 2024-03-29T11:18:02.376 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:02.376 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:02.376 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 30 M 2024-03-29T11:18:02.376 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 107 M 2024-03-29T11:18:02.376 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:02.530 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================= 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================= 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:18:02.534 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:18:02.535 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================= 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:Install 36 Packages 2024-03-29T11:18:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:02.537 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 30 M 2024-03-29T11:18:02.537 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 107 M 2024-03-29T11:18:02.537 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:03.212 INFO:teuthology.orchestra.run.smithi152.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 32 kB/s | 6.6 kB 00:00 2024-03-29T11:18:03.405 INFO:teuthology.orchestra.run.smithi138.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-29T11:18:03.495 INFO:teuthology.orchestra.run.smithi152.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.1 MB/s | 1.5 MB 00:00 2024-03-29T11:18:03.612 INFO:teuthology.orchestra.run.smithi152.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.5 MB/s | 2.1 MB 00:00 2024-03-29T11:18:03.679 INFO:teuthology.orchestra.run.smithi152.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T11:18:03.795 INFO:teuthology.orchestra.run.smithi152.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:18:03.838 INFO:teuthology.orchestra.run.smithi152.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:18:03.871 INFO:teuthology.orchestra.run.smithi152.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-29T11:18:03.905 INFO:teuthology.orchestra.run.smithi152.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:18:03.938 INFO:teuthology.orchestra.run.smithi152.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:18:03.971 INFO:teuthology.orchestra.run.smithi138.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-03-29T11:18:03.980 INFO:teuthology.orchestra.run.smithi152.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:18:04.014 INFO:teuthology.orchestra.run.smithi138.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 2.1 MB 00:00 2024-03-29T11:18:04.047 INFO:teuthology.orchestra.run.smithi152.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:18:04.081 INFO:teuthology.orchestra.run.smithi152.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:18:04.114 INFO:teuthology.orchestra.run.smithi152.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:18:04.147 INFO:teuthology.orchestra.run.smithi138.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.8 MB/s | 247 kB 00:00 2024-03-29T11:18:04.148 INFO:teuthology.orchestra.run.smithi152.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:18:04.264 INFO:teuthology.orchestra.run.smithi138.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:18:04.290 INFO:teuthology.orchestra.run.smithi152.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 883 kB/s | 125 kB 00:00 2024-03-29T11:18:04.306 INFO:teuthology.orchestra.run.smithi138.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:18:04.340 INFO:teuthology.orchestra.run.smithi138.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 942 kB/s | 32 kB 00:00 2024-03-29T11:18:04.374 INFO:teuthology.orchestra.run.smithi152.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-29T11:18:04.374 INFO:teuthology.orchestra.run.smithi138.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:18:04.407 INFO:teuthology.orchestra.run.smithi138.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:18:04.450 INFO:teuthology.orchestra.run.smithi138.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:18:04.517 INFO:teuthology.orchestra.run.smithi138.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:18:04.550 INFO:teuthology.orchestra.run.smithi138.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:18:04.584 INFO:teuthology.orchestra.run.smithi138.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:18:04.591 INFO:teuthology.orchestra.run.smithi152.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 697 kB/s | 151 kB 00:00 2024-03-29T11:18:04.641 INFO:teuthology.orchestra.run.smithi152.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 874 kB/s | 43 kB 00:00 2024-03-29T11:18:04.668 INFO:teuthology.orchestra.run.smithi138.stdout:(14/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.7 MB/s | 4.7 MB 00:01 2024-03-29T11:18:04.693 INFO:teuthology.orchestra.run.smithi138.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.7 MB/s | 187 kB 00:00 2024-03-29T11:18:04.716 INFO:teuthology.orchestra.run.smithi152.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-03-29T11:18:04.808 INFO:teuthology.orchestra.run.smithi152.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:18:04.835 INFO:teuthology.orchestra.run.smithi138.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-03-29T11:18:04.858 INFO:teuthology.orchestra.run.smithi152.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-29T11:18:04.908 INFO:teuthology.orchestra.run.smithi152.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 355 kB/s | 18 kB 00:00 2024-03-29T11:18:04.959 INFO:teuthology.orchestra.run.smithi152.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-29T11:18:05.009 INFO:teuthology.orchestra.run.smithi152.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-29T11:18:05.036 INFO:teuthology.orchestra.run.smithi138.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 755 kB/s | 151 kB 00:00 2024-03-29T11:18:05.086 INFO:teuthology.orchestra.run.smithi138.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 886 kB/s | 43 kB 00:00 2024-03-29T11:18:05.170 INFO:teuthology.orchestra.run.smithi138.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-29T11:18:05.176 INFO:teuthology.orchestra.run.smithi152.stdout:(25/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 10 MB/s | 17 MB 00:01 2024-03-29T11:18:05.193 INFO:teuthology.orchestra.run.smithi152.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 107 kB/s | 20 kB 00:00 2024-03-29T11:18:05.252 INFO:teuthology.orchestra.run.smithi152.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-29T11:18:05.262 INFO:teuthology.orchestra.run.smithi138.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:18:05.287 INFO:teuthology.orchestra.run.smithi138.stdout:(21/36): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-03-29T11:18:05.312 INFO:teuthology.orchestra.run.smithi138.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-03-29T11:18:05.318 INFO:teuthology.orchestra.run.smithi152.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:18:05.335 INFO:teuthology.orchestra.run.smithi152.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 292 kB/s | 46 kB 00:00 2024-03-29T11:18:05.363 INFO:teuthology.orchestra.run.smithi138.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-03-29T11:18:05.369 INFO:teuthology.orchestra.run.smithi152.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-03-29T11:18:05.397 INFO:teuthology.orchestra.run.smithi138.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-03-29T11:18:05.419 INFO:teuthology.orchestra.run.smithi152.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-03-29T11:18:05.422 INFO:teuthology.orchestra.run.smithi138.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-03-29T11:18:05.453 INFO:teuthology.orchestra.run.smithi152.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 767 kB/s | 90 kB 00:00 2024-03-29T11:18:05.464 INFO:teuthology.orchestra.run.smithi138.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-29T11:18:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-29T11:18:05.489 INFO:teuthology.orchestra.run.smithi138.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 689 kB/s | 46 kB 00:00 2024-03-29T11:18:05.556 INFO:teuthology.orchestra.run.smithi138.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:18:05.578 INFO:teuthology.orchestra.run.smithi152.stdout:(34/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.0 MB/s | 4.7 MB 00:02 2024-03-29T11:18:05.581 INFO:teuthology.orchestra.run.smithi138.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 671 kB/s | 79 kB 00:00 2024-03-29T11:18:05.603 INFO:teuthology.orchestra.run.smithi152.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-03-29T11:18:05.607 INFO:teuthology.orchestra.run.smithi138.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 325 kB/s | 16 kB 00:00 2024-03-29T11:18:05.637 INFO:teuthology.orchestra.run.smithi152.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-03-29T11:18:05.641 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:05.641 INFO:teuthology.orchestra.run.smithi152.stdout:Total 9.6 MB/s | 30 MB 00:03 2024-03-29T11:18:05.657 INFO:teuthology.orchestra.run.smithi138.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T11:18:05.664 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:05.683 INFO:teuthology.orchestra.run.smithi138.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-29T11:18:05.704 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:05.704 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:05.750 INFO:teuthology.orchestra.run.smithi138.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2024-03-29T11:18:05.792 INFO:teuthology.orchestra.run.smithi138.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-29T11:18:05.817 INFO:teuthology.orchestra.run.smithi138.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 298 kB/s | 20 kB 00:00 2024-03-29T11:18:06.110 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:06.111 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:06.334 INFO:teuthology.orchestra.run.smithi138.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 7.1 MB/s | 17 MB 00:02 2024-03-29T11:18:06.338 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:06.338 INFO:teuthology.orchestra.run.smithi138.stdout:Total 7.5 MB/s | 30 MB 00:03 2024-03-29T11:18:06.360 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:06.401 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:06.401 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:06.809 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:06.809 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:07.458 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:07.506 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:18:07.540 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:18:07.567 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:18:07.638 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:18:07.674 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:18:07.707 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:18:07.800 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:18:07.902 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:18:07.943 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:18:08.014 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:18:08.068 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:18:08.107 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:18:08.155 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:08.155 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:18:08.210 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:18:08.212 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:18:08.248 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:18:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:18:08.352 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:18:08.397 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:18:08.397 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:18:08.430 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:18:08.443 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:18:08.482 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:18:08.516 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:18:08.521 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:18:08.547 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:18:08.600 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:18:08.624 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:18:08.650 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:18:08.657 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:08.690 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:18:08.722 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:18:08.730 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:18:08.766 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:18:08.784 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:18:08.800 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:18:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:18:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:18:08.888 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:18:08.922 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:18:08.927 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:18:08.995 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:18:09.113 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:18:09.132 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:18:09.170 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:18:09.200 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:18:09.235 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:18:09.249 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:09.266 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:18:09.344 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:18:09.376 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:09.406 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:18:09.410 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:18:09.435 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:18:09.460 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:09.460 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:09.461 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:18:09.461 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:09.461 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:09.461 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:18:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:18:09.626 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:18:09.676 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:18:09.749 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:18:09.888 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:10.169 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:18:10.199 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:10.225 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:10.225 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:10.225 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:18:10.225 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:10.225 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:10.225 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:10.894 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:10.925 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:10.925 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:10.925 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:18:10.925 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:10.925 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:10.925 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:11.370 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:11.393 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:11.393 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:11.393 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:18:11.393 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:11.393 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:11.393 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:11.612 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:11.635 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:11.636 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:11.636 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:18:11.636 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:11.636 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:11.636 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:11.677 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:11.677 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:18:11.677 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:11.677 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:11.702 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:12.127 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:12.150 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:12.150 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:12.150 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:18:12.150 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:12.151 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:12.151 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:12.361 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:12.384 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:12.384 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:12.384 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:18:12.384 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:12.384 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:12.384 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:12.458 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:13.568 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:13.568 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:18:13.568 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:18:13.568 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:18:13.568 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:18:13.569 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:18:13.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:18:13.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:18:13.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:13.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:18:13.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:18:13.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:18:13.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:18:13.571 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:18:13.571 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:18:13.571 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:18:13.571 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:18:13.571 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:13.572 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:18:13.572 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:18:13.572 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:18:13.572 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:18:13.572 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:18:14.223 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:14.223 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:18:14.226 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:18:14.226 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:18:14.226 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:18:14.488 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:18:14.488 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:14.488 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:14.488 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:14.488 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:14.488 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:14.488 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:18:14.489 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:18:14.490 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:14.491 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:14.777 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-03-29T11:18:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:18:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:15.056 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:18:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:18:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:15.059 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:15.348 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:15.430 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-base 2024-03-29T11:18:15.447 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:15.500 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:15.501 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:18:15.501 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:15.569 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-03-29T11:18:16.004 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:16.103 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:16.132 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:16.155 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:16.156 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:18:16.156 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:16.230 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install cephadm 2024-03-29T11:18:16.283 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:16.284 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:16.284 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:16.284 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:16.284 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:16.284 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:18:16.284 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:16.285 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:16.285 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:16.285 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-29T11:18:16.285 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:16.285 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 747 k 2024-03-29T11:18:16.285 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 754 k 2024-03-29T11:18:16.285 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:16.793 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:16.943 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:16.944 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 747 k 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 754 k 2024-03-29T11:18:16.945 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:17.473 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 629 kB/s | 747 kB 00:01 2024-03-29T11:18:17.473 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:17.474 INFO:teuthology.orchestra.run.smithi152.stdout:Total 628 kB/s | 747 kB 00:01 2024-03-29T11:18:17.474 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:17.480 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:17.480 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:17.486 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:17.486 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:17.564 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:17.780 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 896 kB/s | 747 kB 00:00 2024-03-29T11:18:17.781 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:17.781 INFO:teuthology.orchestra.run.smithi138.stdout:Total 894 kB/s | 747 kB 00:00 2024-03-29T11:18:17.781 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:17.786 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:17.786 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:17.792 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:17.792 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:17.866 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:17.952 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.062 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.339 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.427 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.498 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.791 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.791 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:18.791 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:18.792 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:18.792 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:18.792 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.995 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:18:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:19.087 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:19.087 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:19.087 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:19.087 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:19.087 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:19.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:18:19.572 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:19.722 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:19.722 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:19.722 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-29T11:18:19.722 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:19.722 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:19.723 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:18:19.723 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:19.723 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:19.723 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:19.723 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-29T11:18:19.723 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:19.726 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 145 k 2024-03-29T11:18:19.726 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 436 k 2024-03-29T11:18:19.726 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:19.882 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:20.037 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:20.038 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:20.038 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-29T11:18:20.038 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:20.038 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:20.038 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 145 k 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 436 k 2024-03-29T11:18:20.039 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:20.086 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 401 kB/s | 145 kB 00:00 2024-03-29T11:18:20.087 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:20.087 INFO:teuthology.orchestra.run.smithi152.stdout:Total 399 kB/s | 145 kB 00:00 2024-03-29T11:18:20.087 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:20.095 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:20.095 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:20.147 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:20.148 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:20.284 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:20.345 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:20.370 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:20.370 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:20.370 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:18:20.370 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:20.431 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 371 kB/s | 145 kB 00:00 2024-03-29T11:18:20.431 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:20.432 INFO:teuthology.orchestra.run.smithi138.stdout:Total 370 kB/s | 145 kB 00:00 2024-03-29T11:18:20.432 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:20.439 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:20.439 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:20.493 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:20.493 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:20.707 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:20.733 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:20.733 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:20.733 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:18:20.733 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:21.418 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:21.418 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:21.418 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:21.418 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:21.418 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:21.418 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:21.650 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-03-29T11:18:21.737 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:21.809 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:21.809 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:21.809 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:21.810 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:21.810 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:21.979 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr 2024-03-29T11:18:22.229 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:22.329 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:22.381 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:22.382 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:18:22.382 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:22.444 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:18:22.545 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:22.648 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:22.700 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:22.701 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:18:22.701 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:22.773 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:18:23.011 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:23.161 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:23.162 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-03-29T11:18:23.162 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:23.162 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:18:23.163 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:23.164 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:23.164 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-03-29T11:18:23.164 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-03-29T11:18:23.164 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:23.165 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 6.9 M 2024-03-29T11:18:23.165 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-03-29T11:18:23.165 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:23.333 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:23.484 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:23.485 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-29T11:18:23.485 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:23.485 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-29T11:18:23.485 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:23.486 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:18:23.486 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:18:23.486 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:18:23.486 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:18:23.486 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:18:23.487 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:18:23.487 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:18:23.487 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:18:23.487 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:18:23.487 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:18:23.487 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:18:23.488 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:18:23.488 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:18:23.488 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:23.488 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:23.488 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-29T11:18:23.488 INFO:teuthology.orchestra.run.smithi138.stdout:Install 12 Packages 2024-03-29T11:18:23.489 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:23.490 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 6.9 M 2024-03-29T11:18:23.490 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 26 M 2024-03-29T11:18:23.490 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:23.631 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 58 kB/s | 15 kB 00:00 2024-03-29T11:18:23.697 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 95 kB/s | 31 kB 00:00 2024-03-29T11:18:23.773 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.2 MB/s | 238 kB 00:00 2024-03-29T11:18:23.815 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-29T11:18:23.948 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-03-29T11:18:24.054 INFO:teuthology.orchestra.run.smithi138.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 140 kB/s | 31 kB 00:00 2024-03-29T11:18:24.146 INFO:teuthology.orchestra.run.smithi138.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 48 kB/s | 15 kB 00:00 2024-03-29T11:18:24.205 INFO:teuthology.orchestra.run.smithi138.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-29T11:18:24.230 INFO:teuthology.orchestra.run.smithi138.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T11:18:24.299 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 55 kB/s | 19 kB 00:00 2024-03-29T11:18:24.414 INFO:teuthology.orchestra.run.smithi138.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-03-29T11:18:24.491 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 811 kB/s | 548 kB 00:00 2024-03-29T11:18:24.599 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-29T11:18:24.601 INFO:teuthology.orchestra.run.smithi138.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 104 kB/s | 19 kB 00:00 2024-03-29T11:18:24.631 INFO:teuthology.orchestra.run.smithi138.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-29T11:18:24.699 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-29T11:18:24.781 INFO:teuthology.orchestra.run.smithi138.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-29T11:18:24.800 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.2 MB/s | 1.8 MB 00:01 2024-03-29T11:18:24.806 INFO:teuthology.orchestra.run.smithi138.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 823 kB/s | 144 kB 00:00 2024-03-29T11:18:24.825 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-29T11:18:24.849 INFO:teuthology.orchestra.run.smithi138.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 459 kB/s | 31 kB 00:00 2024-03-29T11:18:24.917 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-29T11:18:24.921 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:24.921 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.9 MB/s | 6.9 MB 00:01 2024-03-29T11:18:24.924 INFO:teuthology.orchestra.run.smithi138.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T11:18:24.940 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:24.958 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:24.958 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:25.138 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:25.138 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:25.499 INFO:teuthology.orchestra.run.smithi138.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.1 MB/s | 1.8 MB 00:01 2024-03-29T11:18:25.502 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:25.502 INFO:teuthology.orchestra.run.smithi138.stdout:Total 3.4 MB/s | 6.9 MB 00:02 2024-03-29T11:18:25.521 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:25.538 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:25.538 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:25.651 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:25.703 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:25.703 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:25.726 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:18:25.769 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:18:25.816 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:18:25.855 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:18:25.950 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:18:26.071 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:18:26.158 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:18:26.225 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:18:26.235 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:26.254 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:26.285 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:18:26.326 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:18:26.369 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:18:26.446 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:18:26.527 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:18:26.550 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:18:26.583 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:26.672 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:18:26.774 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:18:26.819 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:18:26.848 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:26.880 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:18:27.123 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:18:27.163 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:18:27.464 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:18:27.465 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:18:27.465 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:27.465 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:18:27.465 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:27.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:27.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:27.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:27.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:27.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:27.801 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:27.801 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:28.060 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:18:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:18:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:18:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:18:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:18:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:18:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:18:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:18:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:18:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:18:28.444 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:28.444 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:28.444 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:28.444 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:28.444 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:28.444 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:28.444 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:28.445 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:28.623 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:28.726 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:28.778 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:28.779 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-03-29T11:18:28.780 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:28.781 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 37 M 2024-03-29T11:18:28.781 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 209 M 2024-03-29T11:18:28.781 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:29.298 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 247 kB/s | 32 kB 00:00 2024-03-29T11:18:29.298 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:29.331 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-29T11:18:29.406 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T11:18:29.453 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 285 kB/s | 34 kB 00:00 2024-03-29T11:18:29.458 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout:Install 12 Packages 2024-03-29T11:18:29.461 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:29.462 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 37 M 2024-03-29T11:18:29.462 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 209 M 2024-03-29T11:18:29.462 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:29.648 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:18:29.724 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:18:29.749 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T11:18:30.133 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T11:18:30.225 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-29T11:18:30.228 INFO:teuthology.orchestra.run.smithi138.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 32 kB 00:00 2024-03-29T11:18:30.262 INFO:teuthology.orchestra.run.smithi138.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 520 kB/s | 17 kB 00:00 2024-03-29T11:18:30.312 INFO:teuthology.orchestra.run.smithi138.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 711 kB/s | 34 kB 00:00 2024-03-29T11:18:30.333 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 977 kB/s | 195 kB 00:00 2024-03-29T11:18:30.370 INFO:teuthology.orchestra.run.smithi138.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:18:30.409 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-29T11:18:30.563 INFO:teuthology.orchestra.run.smithi138.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T11:18:30.630 INFO:teuthology.orchestra.run.smithi138.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:18:30.655 INFO:teuthology.orchestra.run.smithi138.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:18:30.972 INFO:teuthology.orchestra.run.smithi138.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-29T11:18:30.992 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.1 MB/s | 7.4 MB 00:01 2024-03-29T11:18:30.994 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:30.995 INFO:teuthology.orchestra.run.smithi152.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T11:18:31.037 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:31.061 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:31.061 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:31.106 INFO:teuthology.orchestra.run.smithi138.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-29T11:18:31.131 INFO:teuthology.orchestra.run.smithi138.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-29T11:18:31.273 INFO:teuthology.orchestra.run.smithi138.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-29T11:18:31.418 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:31.418 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:31.956 INFO:teuthology.orchestra.run.smithi138.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.0 MB/s | 7.4 MB 00:01 2024-03-29T11:18:31.959 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:31.959 INFO:teuthology.orchestra.run.smithi138.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-29T11:18:32.000 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:32.024 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:32.025 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:32.131 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:32.207 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:32.238 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:32.379 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:32.380 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:32.416 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:32.598 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:33.077 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:33.147 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:33.178 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:33.274 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:33.315 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:33.365 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:33.485 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:33.492 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:33.522 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:33.604 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:34.253 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:34.362 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:34.744 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:35.584 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:35.762 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:35.788 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:36.613 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:36.635 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:38.469 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:38.469 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:38.469 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:38.470 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:38.875 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:38.875 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:38.875 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:38.876 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:39.251 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:39.275 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:39.749 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:39.824 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:39.976 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:39.977 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-29T11:18:39.977 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:39.978 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-29T11:18:39.979 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-03-29T11:18:39.980 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:39.980 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.0 M 2024-03-29T11:18:39.980 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-03-29T11:18:39.980 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:40.146 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:40.154 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 144 kB/s | 18 kB 00:00 2024-03-29T11:18:40.179 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 162 kB/s | 24 kB 00:00 2024-03-29T11:18:40.230 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:18:40.255 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:18:40.280 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:40.297 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:18:40.314 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 183 kB/s | 52 kB 00:00 2024-03-29T11:18:40.556 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-03-29T11:18:40.573 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-29T11:18:40.723 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:40.733 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 369 kB/s | 59 kB 00:00 2024-03-29T11:18:40.759 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 456 kB/s | 203 kB 00:00 2024-03-29T11:18:40.859 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2024-03-29T11:18:40.877 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:40.878 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-29T11:18:40.878 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:40.878 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-29T11:18:40.878 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:40.878 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:40.878 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:40.879 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:40.880 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:40.880 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-29T11:18:40.880 INFO:teuthology.orchestra.run.smithi138.stdout:Install 13 Packages 2024-03-29T11:18:40.880 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:40.880 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 2.0 M 2024-03-29T11:18:40.880 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 26 M 2024-03-29T11:18:40.880 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:40.892 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-29T11:18:40.896 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:40.896 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:18:40.897 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:40.909 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:40.910 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:41.001 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:41.001 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:41.050 INFO:teuthology.orchestra.run.smithi138.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 24 kB 00:00 2024-03-29T11:18:41.067 INFO:teuthology.orchestra.run.smithi138.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 18 kB 00:00 2024-03-29T11:18:41.118 INFO:teuthology.orchestra.run.smithi138.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:18:41.143 INFO:teuthology.orchestra.run.smithi138.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:18:41.168 INFO:teuthology.orchestra.run.smithi138.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:41.193 INFO:teuthology.orchestra.run.smithi138.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-29T11:18:41.219 INFO:teuthology.orchestra.run.smithi138.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 182 kB/s | 52 kB 00:00 2024-03-29T11:18:41.436 INFO:teuthology.orchestra.run.smithi138.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-29T11:18:41.462 INFO:teuthology.orchestra.run.smithi138.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-29T11:18:41.518 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:41.612 INFO:teuthology.orchestra.run.smithi138.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 516 kB/s | 203 kB 00:00 2024-03-29T11:18:41.637 INFO:teuthology.orchestra.run.smithi138.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-29T11:18:41.681 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:41.735 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:41.737 INFO:teuthology.orchestra.run.smithi138.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-03-29T11:18:41.799 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:41.804 INFO:teuthology.orchestra.run.smithi138.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-03-29T11:18:41.808 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:41.809 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:18:41.809 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:41.822 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:41.822 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:41.836 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:41.877 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:41.912 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:41.914 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:41.971 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:42.079 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:42.128 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:42.434 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:42.608 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:42.657 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:42.711 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:42.749 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:42.790 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:42.855 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:42.883 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:42.900 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:42.934 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:42.984 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:42.984 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:43.012 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:43.042 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:43.772 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:43.816 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:43.849 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:43.878 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:43.878 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:43.878 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:43.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:43.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:43.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:43.897 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:43.920 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:44.233 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:44.235 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:44.493 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:44.673 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:44.673 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:44.673 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:44.673 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:44.673 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:45.036 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:45.036 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:45.036 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:45.036 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:45.036 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:45.036 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:45.064 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:45.215 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:45.216 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:45.217 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:45.217 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:45.217 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:45.217 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:45.217 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-03-29T11:18:45.217 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-03-29T11:18:45.217 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:45.218 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 6.8 M 2024-03-29T11:18:45.218 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-03-29T11:18:45.218 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:45.317 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:45.886 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:45.910 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 534 kB/s | 138 kB 00:00 2024-03-29T11:18:45.943 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 850 kB/s | 248 kB 00:00 2024-03-29T11:18:46.037 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:46.038 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-29T11:18:46.038 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:46.038 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-29T11:18:46.038 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:46.038 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-03-29T11:18:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:46.040 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 6.8 M 2024-03-29T11:18:46.040 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 33 M 2024-03-29T11:18:46.040 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:46.118 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-29T11:18:46.144 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 331 kB/s | 58 kB 00:00 2024-03-29T11:18:46.436 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2024-03-29T11:18:46.461 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 995 kB/s | 548 kB 00:00 2024-03-29T11:18:46.464 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:46.464 INFO:teuthology.orchestra.run.smithi152.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-29T11:18:46.477 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:46.487 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:46.488 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:46.536 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:18:46.576 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:46.576 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:46.586 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 413 kB/s | 138 kB 00:00 2024-03-29T11:18:46.889 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:46.919 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-29T11:18:47.020 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 508 kB/s | 51 kB 00:00 2024-03-29T11:18:47.052 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:47.053 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:18:47.104 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:47.246 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 250 kB/s | 248 kB 00:00 2024-03-29T11:18:47.249 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:47.249 INFO:teuthology.orchestra.run.smithi138.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-03-29T11:18:47.263 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:47.273 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:47.273 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:47.365 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:47.366 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:47.624 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:47.664 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:47.706 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:47.768 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:47.791 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:47.821 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:47.872 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:48.383 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:48.467 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:48.617 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:48.617 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:48.617 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:48.617 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:48.617 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:48.617 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:48.914 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:48.915 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:49.149 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-03-29T11:18:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:49.664 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:49.664 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:49.664 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:49.664 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:49.664 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:49.665 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:49.713 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:13 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:49.866 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:49.866 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:49.866 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-03-29T11:18:49.867 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:49.868 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 964 k 2024-03-29T11:18:49.868 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 2.8 M 2024-03-29T11:18:49.868 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:49.940 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-fuse 2024-03-29T11:18:50.037 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 467 kB/s | 79 kB 00:00 2024-03-29T11:18:50.320 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T11:18:50.321 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:50.321 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T11:18:50.322 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:50.341 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:50.342 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:50.415 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:50.415 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:50.502 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:50.596 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:50.652 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:50.653 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-03-29T11:18:50.654 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:50.654 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 964 k 2024-03-29T11:18:50.654 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 2.8 M 2024-03-29T11:18:50.654 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:50.746 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:50.774 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 656 kB/s | 79 kB 00:00 2024-03-29T11:18:50.819 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:51.091 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 885 kB 00:00 2024-03-29T11:18:51.092 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:51.093 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T11:18:51.093 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:51.113 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:51.113 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:51.186 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:51.186 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:51.363 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:51.567 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:51.567 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:51.897 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:51.898 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:51.898 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:51.898 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:51.898 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:51.898 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:52.225 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-volume 2024-03-29T11:18:52.247 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:52.247 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:52.532 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:52.532 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:52.805 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:52.839 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-volume 2024-03-29T11:18:52.959 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:52.959 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:52.959 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-29T11:18:52.959 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:52.959 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:52.959 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:52.960 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 332 k 2024-03-29T11:18:52.961 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 1.5 M 2024-03-29T11:18:52.961 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:53.090 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 530 kB/s | 69 kB 00:00 2024-03-29T11:18:53.316 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 743 kB/s | 264 kB 00:00 2024-03-29T11:18:53.317 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:53.317 INFO:teuthology.orchestra.run.smithi152.stdout:Total 930 kB/s | 332 kB 00:00 2024-03-29T11:18:53.317 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:53.326 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:53.326 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:53.360 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:53.361 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:53.419 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:53.507 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:53.578 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:53.578 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:53.578 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-03-29T11:18:53.579 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:53.580 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 332 k 2024-03-29T11:18:53.580 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.5 M 2024-03-29T11:18:53.580 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:53.681 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:53.700 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 574 kB/s | 69 kB 00:00 2024-03-29T11:18:53.718 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:53.734 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:53.734 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:53.734 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:53.734 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:54.041 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 571 kB/s | 264 kB 00:00 2024-03-29T11:18:54.043 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:54.043 INFO:teuthology.orchestra.run.smithi138.stdout:Total 716 kB/s | 332 kB 00:00 2024-03-29T11:18:54.043 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:54.052 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:54.052 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:54.088 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:54.088 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:54.242 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:54.416 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:54.442 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:54.461 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:54.478 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:54.478 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:54.478 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:54.478 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:54.751 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:54.751 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:54.751 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:54.751 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:54.751 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:54.751 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:54.751 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:55.086 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-03-29T11:18:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:55.581 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:55.581 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:55.581 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:55.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:55.581 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:55.581 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:55.582 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:55.665 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:55.815 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:55.816 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:55.816 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-29T11:18:55.816 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:55.816 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:55.816 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:55.816 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:55.817 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:55.817 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:55.817 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-29T11:18:55.817 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:55.817 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 127 k 2024-03-29T11:18:55.817 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 456 k 2024-03-29T11:18:55.817 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:55.917 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados-devel 2024-03-29T11:18:56.029 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 598 kB/s | 127 kB 00:00 2024-03-29T11:18:56.030 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:56.030 INFO:teuthology.orchestra.run.smithi152.stdout:Total 594 kB/s | 127 kB 00:00 2024-03-29T11:18:56.030 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:56.035 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:56.036 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:56.091 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:56.091 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:56.243 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:56.497 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:56.647 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:56.648 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:56.648 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-29T11:18:56.648 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:56.648 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:18:56.648 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:56.648 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:56.649 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:18:56.649 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:18:56.649 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-29T11:18:56.649 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:56.649 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 127 k 2024-03-29T11:18:56.649 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 456 k 2024-03-29T11:18:56.649 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:18:56.826 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:57.114 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:57.159 INFO:teuthology.orchestra.run.smithi138.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 248 kB/s | 127 kB 00:00 2024-03-29T11:18:57.160 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:57.160 INFO:teuthology.orchestra.run.smithi138.stdout:Total 248 kB/s | 127 kB 00:00 2024-03-29T11:18:57.160 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:18:57.166 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:18:57.166 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:18:57.221 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:18:57.222 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:18:57.351 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-03-29T11:18:57.367 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:18:57.482 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:57.920 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:57.932 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:58.031 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:58.085 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:58.086 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:18:58.086 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:18:58.152 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:58.221 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:58.222 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:58.222 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:18:58.222 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:58.222 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:18:58.222 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:58.488 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs2 2024-03-29T11:18:58.719 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:18:58.871 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:18:58.871 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 31 k 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 140 k 2024-03-29T11:18:58.872 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:18:59.039 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 187 kB/s | 31 kB 00:00 2024-03-29T11:18:59.040 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:59.040 INFO:teuthology.orchestra.run.smithi152.stdout:Total 185 kB/s | 31 kB 00:00 2024-03-29T11:18:59.040 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:18:59.043 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:18:59.044 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:18:59.068 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:18:59.069 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:18:59.070 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:59.173 INFO:teuthology.orchestra.run.smithi138.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:59.192 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:18:59.228 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:18:59.229 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:18:59.229 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:18:59.287 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:59.310 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:59.676 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:59.902 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:18:59.943 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:59.943 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:59.944 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:18:59.944 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:59.944 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:18:59.944 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:00.066 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-29T11:19:00.067 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:00.068 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 31 k 2024-03-29T11:19:00.068 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 140 k 2024-03-29T11:19:00.068 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:19:00.113 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-03-29T11:19:00.275 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 151 kB/s | 31 kB 00:00 2024-03-29T11:19:00.275 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:00.276 INFO:teuthology.orchestra.run.smithi138.stdout:Total 149 kB/s | 31 kB 00:00 2024-03-29T11:19:00.276 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:19:00.280 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:19:00.280 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:19:00.307 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:19:00.308 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:19:00.443 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:19:00.533 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:00.675 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:00.773 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:00.825 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:00.826 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:19:00.826 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:00.878 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-03-29T11:19:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:01.193 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:01.194 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:01.194 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:19:01.194 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:01.194 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:01.194 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:01.348 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados2 2024-03-29T11:19:01.441 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:01.539 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:01.590 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:01.591 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:19:01.591 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:01.643 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-03-29T11:19:01.916 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:02.066 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:02.067 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:19:02.067 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:02.115 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librbd1 2024-03-29T11:19:02.210 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:02.308 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:02.361 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:02.362 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:19:02.362 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:02.410 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-03-29T11:19:02.682 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:02.780 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:02.832 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:02.833 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:19:02.833 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:02.888 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rados 2024-03-29T11:19:02.976 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:03.074 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:03.126 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:03.127 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:19:03.128 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:03.176 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-03-29T11:19:03.457 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:03.557 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:03.609 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:03.610 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:19:03.610 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:03.663 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rgw 2024-03-29T11:19:03.746 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:03.846 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:03.898 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:03.899 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:19:03.899 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:03.950 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-03-29T11:19:04.231 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:04.330 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:04.383 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:04.384 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:19:04.384 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:04.438 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-cephfs 2024-03-29T11:19:04.515 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:04.614 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:04.667 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:04.668 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-03-29T11:19:04.668 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:04.725 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-03-29T11:19:05.003 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:05.102 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:05.153 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:05.154 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:19:05.154 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:05.202 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rbd 2024-03-29T11:19:05.286 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 85 k 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 231 k 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:19:05.767 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:05.921 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:05.922 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-29T11:19:05.922 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:05.941 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 168 kB/s | 85 kB 00:00 2024-03-29T11:19:05.942 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi152.stdout:Total 168 kB/s | 85 kB 00:00 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:19:05.950 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:19:05.950 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:19:05.979 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-fuse 2024-03-29T11:19:06.007 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:19:06.007 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:19:06.142 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:19:06.231 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:06.649 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:06.693 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:06.694 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:06.694 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:06.694 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:06.694 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 85 k 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 231 k 2024-03-29T11:19:06.695 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:19:06.979 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:06.979 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:06.979 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:19:06.979 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:06.980 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:06.980 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:06.987 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 292 kB/s | 85 kB 00:00 2024-03-29T11:19:06.987 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:06.988 INFO:teuthology.orchestra.run.smithi138.stdout:Total 291 kB/s | 85 kB 00:00 2024-03-29T11:19:06.988 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:19:06.995 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:19:06.995 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:19:07.051 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:19:07.051 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:19:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:19:07.280 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-03-29T11:19:07.311 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:07.876 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:07.979 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:07.979 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:07.979 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:19:07.979 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:07.979 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:07.979 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:08.036 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:08.037 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.1 M 2024-03-29T11:19:08.038 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 13 M 2024-03-29T11:19:08.038 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:19:08.214 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-mirror 2024-03-29T11:19:08.793 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:08.951 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:19:08.952 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:08.952 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-29T11:19:08.952 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:08.952 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.1 M 2024-03-29T11:19:08.952 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 13 M 2024-03-29T11:19:08.952 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:19:09.186 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.7 MB/s | 3.1 MB 00:01 2024-03-29T11:19:09.187 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:09.187 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.7 MB/s | 3.1 MB 00:01 2024-03-29T11:19:09.188 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:19:09.197 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:19:09.197 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:19:09.271 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:19:09.271 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:19:09.694 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:19:09.745 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:09.771 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:09.771 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:09.771 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:19:09.771 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:09.771 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:09.771 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:10.317 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.3 MB/s | 3.1 MB 00:01 2024-03-29T11:19:10.318 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:10.318 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.3 MB/s | 3.1 MB 00:01 2024-03-29T11:19:10.318 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:19:10.328 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:19:10.328 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:19:10.401 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:19:10.402 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:19:10.837 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:19:10.901 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:10.926 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:10.926 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:10.926 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:19:10.926 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:10.926 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:10.927 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:10.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:10.953 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:10.953 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:19:10.953 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:10.953 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:10.953 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:11.192 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-03-29T11:19:11.770 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:16:36 AM UTC. 2024-03-29T11:19:11.918 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-29T11:19:11.919 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:11.920 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 171 k 2024-03-29T11:19:11.920 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 494 k 2024-03-29T11:19:11.920 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-29T11:19:12.080 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:12.080 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:12.080 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:19:12.080 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:12.080 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:12.080 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:12.147 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 756 kB/s | 171 kB 00:00 2024-03-29T11:19:12.147 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:12.148 INFO:teuthology.orchestra.run.smithi152.stdout:Total 751 kB/s | 171 kB 00:00 2024-03-29T11:19:12.148 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-29T11:19:12.155 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-29T11:19:12.155 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-29T11:19:12.211 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-29T11:19:12.211 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-29T11:19:12.355 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-nbd 2024-03-29T11:19:12.365 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-29T11:19:12.478 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:12.892 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:12.936 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:16:35 AM UTC. 2024-03-29T11:19:13.085 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-29T11:19:13.087 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:13.087 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:13.087 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:13.088 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 171 k 2024-03-29T11:19:13.089 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 494 k 2024-03-29T11:19:13.089 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-29T11:19:13.299 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 812 kB/s | 171 kB 00:00 2024-03-29T11:19:13.299 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:13.299 INFO:teuthology.orchestra.run.smithi138.stdout:Total 806 kB/s | 171 kB 00:00 2024-03-29T11:19:13.300 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-29T11:19:13.308 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-29T11:19:13.308 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-29T11:19:13.364 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-29T11:19:13.365 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-29T11:19:13.374 DEBUG:teuthology.parallel:result is None 2024-03-29T11:19:13.513 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-29T11:19:13.606 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:14.041 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:14.336 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-29T11:19:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:14.336 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-29T11:19:14.609 DEBUG:teuthology.parallel:result is None 2024-03-29T11:19:14.610 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:19:14.752 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:19:14.772 INFO:teuthology.orchestra.run.smithi138.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:14.773 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:14.773 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:19:14.775 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:19:14.916 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:19:14.941 INFO:teuthology.orchestra.run.smithi152.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:14.941 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:14.941 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:19:14.943 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:19:14.943 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:14.943 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:19:14.972 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:14.972 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:19:15.014 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:19:15.014 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:15.014 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:19:15.041 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:19:15.108 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:15.108 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:19:15.138 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:19:15.207 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:19:15.207 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:15.207 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:19:15.234 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:19:15.300 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:15.301 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:19:15.329 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:19:15.397 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:19:15.397 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:15.398 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:19:15.424 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:19:15.492 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:15.492 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:19:15.519 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:19:15.587 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:19:15.669 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:19:15.670 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /var/log/ceph 2024-03-29T11:19:15.672 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2024-03-29T11:19:15.699 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:19:15.699 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:19:15.739 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:19:15.766 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:19:15.766 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:19:15.806 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:19:15.842 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:19:15.843 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:19:15.843 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622696', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi138.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGpptLcC57lUuS0F5ZaBYanQ/mD3T/64LfMnxx3oNg5dHdHQtLjUnvP8USSCCr61/p4NtkvfPKrD8ypNlloLkw=', 'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAxy1Of+RLhQGS1byxOnbjm9Sof/8KrE4Eb13V1+djlQp3zjp6NipL3pg8z+KxDbrJI5ZuZbWw2almQYuUn85mc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:19:15.843 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:15.873 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:15.888 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:15.928 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:15.955 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:15.955 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:16.007 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:16.007 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:16.061 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-29 11:19:11.202160109 +0000 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-29 11:15:43.636201134 +0000 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-29 11:15:43.636201134 +0000 2024-03-29T11:19:16.062 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-29 11:15:43.636201134 +0000 2024-03-29T11:19:16.062 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:16.127 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-29T11:19:16.127 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-29T11:19:16.127 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000203434 s, 2.5 MB/s 2024-03-29T11:19:16.128 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:16.184 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:16.239 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:19:16.240 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:16.240 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:16.240 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-29 11:19:11.202160109 +0000 2024-03-29T11:19:16.240 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-29 11:15:43.971194217 +0000 2024-03-29T11:19:16.240 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-29 11:15:43.971194217 +0000 2024-03-29T11:19:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-29 11:15:43.971194217 +0000 2024-03-29T11:19:16.240 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:16.305 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-29T11:19:16.305 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-29T11:19:16.305 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000218243 s, 2.3 MB/s 2024-03-29T11:19:16.306 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:16.362 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:16.416 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:16.416 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:16.417 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:19:16.417 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:16.417 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:16.417 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-29 11:19:11.203160088 +0000 2024-03-29T11:19:16.417 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-29 11:15:44.317187073 +0000 2024-03-29T11:19:16.417 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-29 11:15:44.317187073 +0000 2024-03-29T11:19:16.417 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-29 11:15:44.317187073 +0000 2024-03-29T11:19:16.417 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:16.482 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-29T11:19:16.482 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-29T11:19:16.482 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000230088 s, 2.2 MB/s 2024-03-29T11:19:16.483 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:16.539 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-29 11:19:11.203160088 +0000 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-29 11:15:44.651180177 +0000 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-29 11:15:44.651180177 +0000 2024-03-29T11:19:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-29 11:15:44.651180177 +0000 2024-03-29T11:19:16.596 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:16.661 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-29T11:19:16.661 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-29T11:19:16.661 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000207272 s, 2.5 MB/s 2024-03-29T11:19:16.662 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:16.718 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:16.719 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:16.719 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:16.736 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:16.736 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-29 11:19:10.089427827 +0000 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-29 11:15:53.578243342 +0000 2024-03-29T11:19:16.792 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-29 11:15:53.578243342 +0000 2024-03-29T11:19:16.793 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-29 11:15:53.578243342 +0000 2024-03-29T11:19:16.793 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:16.860 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-29T11:19:16.860 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-29T11:19:16.860 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000284871 s, 1.8 MB/s 2024-03-29T11:19:16.861 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:16.917 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:16.973 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-29 11:19:10.089427827 +0000 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-29 11:15:53.933238821 +0000 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-29 11:15:53.933238821 +0000 2024-03-29T11:19:16.974 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-29 11:15:53.933238821 +0000 2024-03-29T11:19:16.974 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:17.041 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-29T11:19:17.041 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-29T11:19:17.041 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000191829 s, 2.7 MB/s 2024-03-29T11:19:17.042 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:17.098 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-29 11:19:10.090427815 +0000 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-29 11:15:54.268234555 +0000 2024-03-29T11:19:17.153 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-29 11:15:54.268234555 +0000 2024-03-29T11:19:17.154 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-29 11:15:54.268234555 +0000 2024-03-29T11:19:17.154 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:17.219 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-29T11:19:17.219 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-29T11:19:17.219 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000189221 s, 2.7 MB/s 2024-03-29T11:19:17.220 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:17.275 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:17.329 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-29 11:19:10.091427802 +0000 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-29 11:15:54.603230290 +0000 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-29 11:15:54.603230290 +0000 2024-03-29T11:19:17.330 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-29 11:15:54.603230290 +0000 2024-03-29T11:19:17.330 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-29T11:19:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-29T11:19:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000206249 s, 2.5 MB/s 2024-03-29T11:19:17.396 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:17.452 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:17.452 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:17.452 INFO:tasks.ceph:Generating config... 2024-03-29T11:19:17.453 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:19:17.453 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:19:17.453 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:19:17.453 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:19:17.453 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:19:17.454 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:19:17.454 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:19:17.454 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:19:17.454 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:19:17.454 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:19:17.454 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:19:17.455 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:19:17.455 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:19:17.456 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:19:17.456 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:19:17.456 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:17.510 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:19:17.513 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:19:17.559 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:17.587 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.138:6789]'), ('mon.c', '[v1:172.21.15.138:6790]'), ('mon.b', '[v1:172.21.15.152:6789]')] 2024-03-29T11:19:17.587 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.138:6789],[v1:172.21.15.138:6790],[v1:172.21.15.152:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:19:17.588 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:17.588 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:17.642 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.138:6789]' --addv c '[v1:172.21.15.138:6790]' --addv b '[v1:172.21.15.152:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: generated fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:setting min_mon_release = quincy 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:epoch 0 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:min_mon_release 17 (quincy) 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:election_strategy: 1 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:17.728 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:19:17.729 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:17.743 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 113c9c6c-c44b-4755-b507-344fd243c0cb... 2024-03-29T11:19:17.744 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:17.785 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:17.832 INFO:teuthology.orchestra.run.smithi138.stdout:[global] 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: chdir = "" 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: auth supported = cephx 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: filestore xattr use omap = true 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: auth debug = true 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on old message = true 2024-03-29T11:19:17.833 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on bug = true 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default size = 2 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow pg remap = true 2024-03-29T11:19:17.834 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on too few osds = false 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: osd default data pool replay window = 5 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.835 INFO:teuthology.orchestra.run.smithi138.stdout: mon allow pool delete = true 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: mon cluster log file level = debug 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: debug asserts on shutdown = true 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: mon health detail to clog = false 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: mon host = "[v1:172.21.15.138:6789],[v1:172.21.15.138:6790],[v1:172.21.15.152:6789]" 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: lockdep = True 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: mon client directed command retry = 5 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: mon election default strategy = 1 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: ms bind msgr2 = False 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject socket failures = 5000 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: ms type = async 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: fsid = 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout:[osd] 2024-03-29T11:19:17.836 INFO:teuthology.orchestra.run.smithi138.stdout: osd journal size = 100 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub max interval = 600 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd recover clone overlap = true 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug shutdown = true 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug op order = true 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug trim objects = true 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd open classes on start = true 2024-03-29T11:19:17.837 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug pg log writeout = true 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: osd map max advance = 10 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: journal zero on create = true 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: bdev debug aio = true 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug misdirected ops = true 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore block size = 96636764160 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:17.838 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore fsck on mount = True 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore zero block detection = True 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluefs = 20 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluestore = 20 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: debug monc = 20 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: debug osd = 20 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: debug rocksdb = 10 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: osd class default list = * 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: osd class load list = * 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: osd client watch timeout = 120 2024-03-29T11:19:17.839 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify missing on start = True 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: osd objectstore = bluestore 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: osd op queue = debug_random 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr] 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: debug mgr = 20 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2024-03-29T11:19:17.840 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout:[mon] 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: debug paxos = 20 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mon data avail warn = 5 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd prime pg temp = true 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.841 INFO:teuthology.orchestra.run.smithi138.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: # 1m isn't quite enough 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:17.842 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: mon scrub interval = 300 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on pool no app = False 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: paxos service trim min = 10 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout:[client] 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: rgw cache enabled = true 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable ops log = true 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable usage log = true 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: debug monc = 20 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: debug objecter = 20 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout: debug rados = 20 2024-03-29T11:19:17.843 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.a] 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.c] 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.b] 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-03-29T11:19:17.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.845 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = "[v1:172.21.15.138:6789],[v1:172.21.15.138:6790],[v1:172.21.15.152:6789]" 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: lockdep = True 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon client directed command retry = 5 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon election default strategy = 1 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: ms bind msgr2 = False 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject socket failures = 5000 2024-03-29T11:19:17.847 INFO:teuthology.orchestra.run.smithi152.stdout: ms type = async 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-03-29T11:19:17.848 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug trim objects = true 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:17.849 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore zero block detection = True 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 20 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 20 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 20 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-03-29T11:19:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 10 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd class default list = * 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd class load list = * 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd client watch timeout = 120 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify missing on start = True 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue = debug_random 2024-03-29T11:19:17.851 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-03-29T11:19:17.852 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon scrub interval = 300 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on pool no app = False 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: paxos service trim min = 10 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-03-29T11:19:17.854 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 20 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: debug objecter = 20 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout: debug rados = 20 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.c] 2024-03-29T11:19:17.855 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.b] 2024-03-29T11:19:17.857 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:19:17.857 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:19:17.952 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:19:17.952 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:17.952 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:19:17.967 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:17.967 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:19:18.021 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:19:18.021 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:18.022 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:18.022 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:18.101 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:18.101 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:18.158 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:19:18.158 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:18.158 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:18.158 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:18.199 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:18.199 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:18.254 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:19:18.254 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:19:18.254 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:18.310 INFO:teuthology.orchestra.run.smithi138.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:18.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:18.379 INFO:teuthology.orchestra.run.smithi152.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:18.382 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:19:18.382 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:19:18.382 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:18.424 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:18.439 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:18.481 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:18.497 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:19:18.497 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:18.497 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:18.524 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:18.524 INFO:tasks.ceph:role: osd.0 2024-03-29T11:19:18.524 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:19:18.525 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:18.599 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:19.095 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-03-29T11:19:19.097 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-03-29T11:19:19.097 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:19.210 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:19.246 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:19.312 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:19.312 INFO:tasks.ceph:role: osd.1 2024-03-29T11:19:19.313 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:19:19.313 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:19.387 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:19.845 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-03-29T11:19:19.847 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-03-29T11:19:19.847 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:19.940 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:19.970 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:20.039 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:20.040 INFO:tasks.ceph:role: osd.2 2024-03-29T11:19:20.040 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:19:20.040 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:20.114 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:20.595 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-03-29T11:19:20.597 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-03-29T11:19:20.597 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:20.704 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:20.736 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:20.803 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:20.804 INFO:tasks.ceph:role: osd.3 2024-03-29T11:19:20.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:19:20.804 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:20.878 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:20.879 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:20.879 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:20.879 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:20.879 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:21.401 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-03-29T11:19:21.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-03-29T11:19:21.403 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:21.545 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:21.582 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:21.704+0000 7f46510ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:19:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:21.704+0000 7f46510ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:19:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:21.704+0000 7f46510ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:21.704+0000 7f46510ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:21.704+0000 7f46510ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:21.705+0000 7f46510ac640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:19:24.532 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:24.563 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:24.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:24.676+0000 7fa042a1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:19:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:24.676+0000 7fa042a1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:19:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:24.676+0000 7fa042a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:24.676+0000 7fa042a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:24.676+0000 7fa042a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:24.677+0000 7fa042a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:19:27.498 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:27.529 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:27.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:27.643+0000 7fe462ca0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:19:27.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:27.643+0000 7fe462ca0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:19:27.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:27.643+0000 7fe462ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:27.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:27.643+0000 7fe462ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:27.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:27.643+0000 7fe462ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:27.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:27.644+0000 7fe462ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:19:30.471 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:30.504 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:30.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:30.619+0000 7f7999a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:19:30.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:30.619+0000 7f7999a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:19:30.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:30.619+0000 7f7999a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:30.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:30.619+0000 7f7999a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:30.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:30.619+0000 7f7999a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:30.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:30.620+0000 7f7999a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:19:33.448 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:33.480 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:33.513 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:33.513 INFO:tasks.ceph:role: osd.4 2024-03-29T11:19:33.513 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:19:33.513 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:33.591 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:33.591 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:33.591 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:33.592 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:33.592 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:33.592 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:33.592 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:33.592 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:33.592 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:33.592 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-03-29T11:19:34.116 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-03-29T11:19:34.116 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:34.263 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:34.300 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:34.369 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:34.369 INFO:tasks.ceph:role: osd.5 2024-03-29T11:19:34.369 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:19:34.369 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:34.444 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:34.445 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:34.445 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:34.886 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-03-29T11:19:34.887 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-03-29T11:19:34.888 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:34.973 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:35.004 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:35.072 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:35.072 INFO:tasks.ceph:role: osd.6 2024-03-29T11:19:35.072 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:19:35.072 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:35.146 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:35.146 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:35.146 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:35.147 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:35.147 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:35.147 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:35.147 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:35.147 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:35.147 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:35.147 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:35.635 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-03-29T11:19:35.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-03-29T11:19:35.637 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:35.781 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:35.813 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:35.880 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:35.880 INFO:tasks.ceph:role: osd.7 2024-03-29T11:19:35.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:19:35.880 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:35.955 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:36.487 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-03-29T11:19:36.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-03-29T11:19:36.489 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:36.641 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:36.675 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:36.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:36.799+0000 7f42c621b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:19:36.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:36.799+0000 7f42c621b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:19:36.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:36.799+0000 7f42c621b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:36.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:36.799+0000 7f42c621b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:36.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:36.799+0000 7f42c621b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:36.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:36.799+0000 7f42c621b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:19:39.633 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:39.664 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:39.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:39.780+0000 7fedbbc15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:19:39.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:39.781+0000 7fedbbc15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:19:39.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:39.781+0000 7fedbbc15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:39.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:39.781+0000 7fedbbc15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:39.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:39.781+0000 7fedbbc15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:39.758 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:39.781+0000 7fedbbc15640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:19:42.615 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:42.646 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:42.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:42.760+0000 7f04dd614640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:19:42.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:42.761+0000 7f04dd614640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:19:42.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:42.761+0000 7f04dd614640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:42.737 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:42.761+0000 7f04dd614640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:42.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:42.761+0000 7f04dd614640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:42.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:42.761+0000 7f04dd614640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:19:45.592 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:45.624 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:45.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:45.736+0000 7fe5a75df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:19:45.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:45.736+0000 7fe5a75df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:19:45.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:45.736+0000 7fe5a75df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:45.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:45.736+0000 7fe5a75df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:45.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:45.736+0000 7fe5a75df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:45.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:45.737+0000 7fe5a75df640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:19:48.557 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:48.588 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:19:48.589 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:48.589 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:19:48.621 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:48.622 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:19:48.694 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:48.694 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:19:48.763 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:48.763 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:19:48.833 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:48.833 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:19:48.901 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:48.901 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:19:48.933 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:48.933 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:19:49.004 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:49.004 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:19:49.074 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:49.074 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:19:49.143 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:49.144 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:19:49.213 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:49.213 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:19:49.230 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:49.230 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:19:49.269 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:19:49.269 DEBUG:teuthology.orchestra.run.smithi138:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:49.273 DEBUG:teuthology.orchestra.run.smithi152:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:49.312 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.y] 2024-03-29T11:19:49.312 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC2owZmepHrExAAcnZywSsnUwEK3reF6MNboQ== 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC5owZmCcgLKhAAqIiZLHJx/FXj8rftzyWoxQ== 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC8owZmqQ5nKBAAX4r3Ff6xPlAryu7R9kagJw== 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC/owZmc/luJhAAIaRjq0qcN+vCXMl5MTIrgA== 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.3] 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQDCowZmum/+JBAAjVMRkffCpxy9pJDipvl3WQ== 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.x] 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC2owZmY6kOGBAARaOx95iXQR5sCXPTV4VFSg== 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.4] 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQDIowZm81uoLxAAHHabXTN7MFmCBmq9x7HOYw== 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.5] 2024-03-29T11:19:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQDLowZmX72PLhAAguTTHKlI18EjsCajM3xLUw== 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.6] 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQDOowZmOtliLRAAd7TDXzgtabAXR0uzVUVEsw== 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.7] 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQDRowZmZxz6KxAAZhI9lVhhR+TrAgb+JdTMOw== 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC2owZmlhHAGhAAoXoM1jj2m8xT0gHJpBqqHA== 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout:[client.1] 2024-03-29T11:19:49.314 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQC2owZmttArHhAANFSJjIfjtpqr4fbXBSld+g== 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.y] 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC2owZmepHrExAAcnZywSsnUwEK3reF6MNboQ== 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC5owZmCcgLKhAAqIiZLHJx/FXj8rftzyWoxQ== 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC8owZmqQ5nKBAAX4r3Ff6xPlAryu7R9kagJw== 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC/owZmc/luJhAAIaRjq0qcN+vCXMl5MTIrgA== 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.3] 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDCowZmum/+JBAAjVMRkffCpxy9pJDipvl3WQ== 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.x] 2024-03-29T11:19:49.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC2owZmY6kOGBAARaOx95iXQR5sCXPTV4VFSg== 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.4] 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDIowZm81uoLxAAHHabXTN7MFmCBmq9x7HOYw== 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.5] 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDLowZmX72PLhAAguTTHKlI18EjsCajM3xLUw== 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.6] 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDOowZmOtliLRAAd7TDXzgtabAXR0uzVUVEsw== 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.7] 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDRowZmZxz6KxAAZhI9lVhhR+TrAgb+JdTMOw== 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout:[client.0] 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC2owZmlhHAGhAAoXoM1jj2m8xT0gHJpBqqHA== 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout:[client.1] 2024-03-29T11:19:49.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC2owZmttArHhAANFSJjIfjtpqr4fbXBSld+g== 2024-03-29T11:19:49.338 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:49.358 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:49.433 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.435 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.487 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.489 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.541 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.543 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.595 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.597 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.648 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:49.651 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:49.702 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.705 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.757 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.759 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.811 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.813 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.864 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.867 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:49.918 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:49.921 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:49.975 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:49.978 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:50.032 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:19:50.032 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:50.059 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:50.156 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:50.183 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:50.250 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:50.346 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:50.374 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:50.401 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:50.501 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:50.528 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:50.530 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:50.582 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:19:50.583 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:19:50.583 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:19:50.585 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:19:50.585 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:19:50.586 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:19:50.587 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:19:50.587 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:19:50.588 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:19:50.625 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:19:50.625 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:19:50.626 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:19:50.626 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:19:50.628 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:19:50.628 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:19:50.628 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:19:50.630 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:19:50.630 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:19:50.631 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:19:50.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.755+0000 7fc439dc6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:50.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.755+0000 7fc439dc6640 10 monclient: build_initial_monmap 2024-03-29T11:19:50.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.755+0000 7fc439dc6640 10 monclient: monmap: 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:50.756424+0000 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:50.756424+0000 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.755+0000 7fc439dc6640 1 Processor -- start 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.755+0000 7fc439dc6640 1 -- start start 2024-03-29T11:19:50.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.755+0000 7fc439dc6640 10 monclient: init 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc43409ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc43409c870 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc4340b4360 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43409dec0 con 0x7fc43409c870 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43409da30 con 0x7fc4340b4360 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4340b39c0 con 0x7fc43409ac40 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc439dc6640 10 monclient(hunting): authenticate will time out at 812.065607s 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc4337fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fc43409ac40 0x7fc4340b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:56800/0 (socket says 172.21.15.138:56800) 2024-03-29T11:19:50.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc432ffd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc43409c870 0x7fc43413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45910/0 (socket says 172.21.15.138:45910) 2024-03-29T11:19:50.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:50.756+0000 7fc4337fe640 1 -- 172.21.15.138:0/2890054533 learned_addr learned my addr 172.21.15.138:0/2890054533 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient(hunting): tick 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 1 -- 172.21.15.138:0/2890054533 >> v1:172.21.15.138:6790/0 conn(0x7fc43409ac40 legacy=0x7fc4340b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 1 -- 172.21.15.138:0/2890054533 >> v1:172.21.15.152:6789/0 conn(0x7fc4340b4360 legacy=0x7fc43414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 1 -- 172.21.15.138:0/2890054533 >> v1:172.21.15.138:6789/0 conn(0x7fc43409c870 legacy=0x7fc43413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc4200036e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc43409ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc42000ab20 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 1 -- 172.21.15.138:0/2890054533 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4340b5160 con 0x7fc4200036e0 2024-03-29T11:19:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 1 -- 172.21.15.138:0/2890054533 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42000f810 con 0x7fc42000ab20 2024-03-29T11:19:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 1 -- 172.21.15.138:0/2890054533 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc420010a50 con 0x7fc43409ac40 2024-03-29T11:19:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:53.757+0000 7fc4317fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834618146 0 0) 0x7fc4340b5160 con 0x7fc4200036e0 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c003520 con 0x7fc4200036e0 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857174116 0 0) 0x7fc420010a50 con 0x7fc43409ac40 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.928+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4340b5160 con 0x7fc43409ac40 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 222830727 0 0) 0x7fc4340b5160 con 0x7fc43409ac40 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 >> v1:172.21.15.152:6789/0 conn(0x7fc42000ab20 legacy=0x7fc42000cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 >> v1:172.21.15.138:6789/0 conn(0x7fc4200036e0 legacy=0x7fc420005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 monclient: found mon.noname-b 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc420011cb0 con 0x7fc43409ac40 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:55.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc4280014f0 con 0x7fc43409ac40 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc439dc6640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc439dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: dump: 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.910 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc439dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4280016c0 con 0x7fc43409ac40 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 1 -- 172.21.15.138:0/2890054533 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc428001a50 con 0x7fc43409ac40 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc439dc6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc439dc6640 4 set_mon_vals no callback set 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.929+0000 7fc4327fc640 10 monclient: dump: 2024-03-29T11:19:55.911 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.930+0000 7fc439dc6640 10 monclient: shutdown 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.930+0000 7fc439dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.930+0000 7fc439dc6640 1 -- 172.21.15.138:0/2890054533 >> v1:172.21.15.138:6790/0 conn(0x7fc43409ac40 legacy=0x7fc420009320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.930+0000 7fc439dc6640 1 -- 172.21.15.138:0/2890054533 shutdown_connections 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.930+0000 7fc439dc6640 1 -- 172.21.15.138:0/2890054533 >> 172.21.15.138:0/2890054533 conn(0x7fc434066b00 msgr2=0x7fc434066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.930+0000 7fc439dc6640 1 -- 172.21.15.138:0/2890054533 shutdown_connections 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.930+0000 7fc439dc6640 1 -- 172.21.15.138:0/2890054533 wait complete. 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.931+0000 7fc439dc6640 10 monclient: build_initial_monmap 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.931+0000 7fc439dc6640 10 monclient: monmap: 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:55.914 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.931+0000 7fc439dc6640 1 librados: starting msgr at 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.931+0000 7fc439dc6640 1 librados: starting objecter 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.931+0000 7fc439dc6640 1 Processor -- start 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 1 -- start start 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 1 librados: setting wanted keys 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 1 librados: calling monclient init 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient: init 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc4200036e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc42000ab20 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc43409ac40 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc434140200 con 0x7fc43409ac40 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4341404e0 con 0x7fc4200036e0 2024-03-29T11:19:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc432ffd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fc42000ab20 0x7fc4341da0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:60204/0 (socket says 172.21.15.138:60204) 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc432ffd640 1 -- 172.21.15.138:0/2413170326 learned_addr learned my addr 172.21.15.138:0/2413170326 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4340b70b0 con 0x7fc42000ab20 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.932+0000 7fc439dc6640 10 monclient(hunting): authenticate will time out at 817.241935s 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 client.?.objecter ms_handle_connect 0x7fc4200036e0 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 client.?.objecter ms_handle_connect 0x7fc42000ab20 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 client.?.objecter ms_handle_connect 0x7fc43409ac40 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948098525 0 0) 0x7fc434140200 con 0x7fc43409ac40 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): my global_id is 4107 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc420010240 con 0x7fc43409ac40 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919407250 0 0) 0x7fc420010240 con 0x7fc43409ac40 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc434140200 con 0x7fc43409ac40 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc42c001a50 con 0x7fc43409ac40 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:55.922 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.933+0000 7fc41bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.934+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806991979 0 0) 0x7fc4340b70b0 con 0x7fc42000ab20 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.934+0000 7fc41bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.934+0000 7fc41bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.934+0000 7fc41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.934+0000 7fc41bfff640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.934+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc420010240 con 0x7fc42000ab20 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118670088 0 0) 0x7fc4341404e0 con 0x7fc4200036e0 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): my global_id is 4099 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4340b70b0 con 0x7fc4200036e0 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3824108901 0 0) 0x7fc420010240 con 0x7fc42000ab20 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4341404e0 con 0x7fc42000ab20 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc428001070 con 0x7fc42000ab20 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1318572228 0 0) 0x7fc4340b70b0 con 0x7fc4200036e0 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc420010240 con 0x7fc4200036e0 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc424001770 con 0x7fc4200036e0 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 752481709 0 0) 0x7fc434140200 con 0x7fc43409ac40 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 >> v1:172.21.15.138:6790/0 conn(0x7fc42000ab20 legacy=0x7fc4341da0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 >> v1:172.21.15.152:6789/0 conn(0x7fc4200036e0 legacy=0x7fc434135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 monclient: found mon.a 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4340b7240 con 0x7fc43409ac40 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc41bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc439dc6640 5 monclient: authenticate success, global_id 4107 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc439dc6640 10 monclient: _renew_subs 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc439dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.935+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc434140be0 con 0x7fc43409ac40 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc439dc6640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc439dc6640 10 monclient: _renew_subs 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc439dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc434141270 con 0x7fc43409ac40 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc439dc6640 1 librados: init done 2024-03-29T11:19:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc42c001fa0 con 0x7fc43409ac40 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc42c003b10 con 0x7fc43409ac40 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 10 monclient: dump: 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:55.928 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc42c004080 con 0x7fc43409ac40 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc431ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3391122577 0 0) 0x7fc42c004330 con 0x7fc43409ac40 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 10 client.4107.objecter ms_dispatch 0x7fc434000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.936+0000 7fc41bfff640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.937+0000 7fc439dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.937+0000 7fc439dc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.937+0000 7fc439dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.937+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc434074bb0 con 0x7fc43409ac40 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.941+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc42c0044f0 con 0x7fc43409ac40 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.941+0000 7fc41bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:55.941+0000 7fc41bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.014+0000 7fc439dc6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.014+0000 7fc439dc6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.014+0000 7fc439dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.014+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc43404f730 con 0x7fc43409ac40 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.016+0000 7fc41bfff640 1 -- 172.21.15.138:0/2413170326 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fc42c0052d0 con 0x7fc43409ac40 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.016+0000 7fc41bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.016+0000 7fc41bfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:adjusted tunables profile to default 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 10 librados: watch_flush enter 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 10 librados: watch_flush exit 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 10 client.4107.objecter successfully canceled tick 2024-03-29T11:19:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 10 monclient: shutdown 2024-03-29T11:19:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 >> v1:172.21.15.138:6789/0 conn(0x7fc43409ac40 legacy=0x7fc4341da7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 shutdown_connections 2024-03-29T11:19:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 >> 172.21.15.138:0/2413170326 conn(0x7fc434066b00 msgr2=0x7fc434069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 shutdown_connections 2024-03-29T11:19:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 1 -- 172.21.15.138:0/2413170326 wait complete. 2024-03-29T11:19:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:56.017+0000 7fc439dc6640 1 librados: shutdown 2024-03-29T11:19:56.005 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:56.005 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:56.006 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:56.006 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:56.033 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:56.033 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:56.100 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:56.100 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:56.167 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-29T11:19:56.167 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:56.235 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:56.235 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:56.262 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:56.262 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:56.329 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:56.329 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:56.396 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-29T11:19:56.396 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:56.410 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:56.433+0000 7f7436c4d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:56.415 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:56.439+0000 7fb653634040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:56.436 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 1d49822d-afc6-4c38-9d56-52b09808838f 0 2024-03-29T11:19:56.468 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:56.490+0000 7f7436c4d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:56.469 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:56.493+0000 7fb653634040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:56.569 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.592+0000 7f96b3154640 10 monclient: get_monmap_and_config 2024-03-29T11:19:56.569 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.592+0000 7f96b3154640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.569 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.592+0000 7f96b3154640 10 monclient: monmap: 2024-03-29T11:19:56.569 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:56.593860+0000 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:56.593860+0000 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:56.570 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 1 Processor -- start 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 1 -- start start 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient: init 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac09ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac09c870 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac0b4360 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac09dec0 con 0x7f96ac0b4360 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac09da30 con 0x7f96ac09c870 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0b39c0 con 0x7f96ac09ac40 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.580 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96b3154640 10 monclient(hunting): authenticate will time out at 816.890754s 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96abfff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f96ac09c870 0x7f96ac13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37354/0 (socket says 172.21.15.152:37354) 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.593+0000 7f96abfff640 1 -- 172.21.15.152:0/2767530178 learned_addr learned my addr 172.21.15.152:0/2767530178 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329390976 0 0) 0x7f96ac09da30 con 0x7f96ac09c870 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c003520 con 0x7f96ac09c870 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127568311 0 0) 0x7f96ac0b39c0 con 0x7f96ac09ac40 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac09da30 con 0x7f96ac09ac40 2024-03-29T11:19:56.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046913971 0 0) 0x7f96a4001490 con 0x7f96ac0b4360 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac0b39c0 con 0x7f96ac0b4360 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922046821 0 0) 0x7f969c003520 con 0x7f96ac09c870 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 >> v1:172.21.15.138:6790/0 conn(0x7f96ac09ac40 legacy=0x7f96ac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 >> v1:172.21.15.138:6789/0 conn(0x7f96ac0b4360 legacy=0x7f96ac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac0b5160 con 0x7f96ac09c870 2024-03-29T11:19:56.582 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9698001690 con 0x7f96ac09c870 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96b3154640 5 monclient: authenticate success, global_id 4102 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96b3154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: dump: 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.583 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96b3154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9698001870 con 0x7f96ac09c870 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96b3154640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 1 -- 172.21.15.152:0/2767530178 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9698001c80 con 0x7f96ac09c870 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96b3154640 4 set_mon_vals no callback set 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.594+0000 7f96ab7fe640 10 monclient: dump: 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.584 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 10 monclient: shutdown 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 1 -- 172.21.15.152:0/2767530178 >> v1:172.21.15.152:6789/0 conn(0x7f96ac09c870 legacy=0x7f96ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 1 -- 172.21.15.152:0/2767530178 shutdown_connections 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 1 -- 172.21.15.152:0/2767530178 >> 172.21.15.152:0/2767530178 conn(0x7f96ac066b00 msgr2=0x7f96ac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.585 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 1 -- 172.21.15.152:0/2767530178 shutdown_connections 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 1 -- 172.21.15.152:0/2767530178 wait complete. 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 10 monclient: monmap: 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 1 librados: starting msgr at 2024-03-29T11:19:56.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.595+0000 7f96b3154640 1 librados: starting objecter 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 1 Processor -- start 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 1 -- start start 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 1 librados: setting wanted keys 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 1 librados: calling monclient init 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient: init 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac09ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac140660 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac140f80 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac1423a0 con 0x7f96ac140f80 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac140140 con 0x7f96ac140660 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0b8040 con 0x7f96ac09ac40 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96b3154640 10 monclient(hunting): authenticate will time out at 816.893788s 2024-03-29T11:19:56.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96abfff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f96ac140660 0x7f96ac0b1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37360/0 (socket says 172.21.15.152:37360) 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.596+0000 7f96abfff640 1 -- 172.21.15.152:0/2788584250 learned_addr learned my addr 172.21.15.152:0/2788584250 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 client.?.objecter ms_handle_connect 0x7f96ac140660 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 client.?.objecter ms_handle_connect 0x7f96ac140f80 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 client.?.objecter ms_handle_connect 0x7f96ac09ac40 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276417043 0 0) 0x7f96ac140140 con 0x7f96ac140660 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9690004eb0 con 0x7f96ac140660 2024-03-29T11:19:56.588 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597573276 0 0) 0x7f96ac1423a0 con 0x7f96ac140f80 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac140140 con 0x7f96ac140f80 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404560135 0 0) 0x7f96ac0b8040 con 0x7f96ac09ac40 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac1423a0 con 0x7f96ac09ac40 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478400043 0 0) 0x7f9690004eb0 con 0x7f96ac140660 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ac0b8040 con 0x7f96ac140660 2024-03-29T11:19:56.589 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9698001bc0 con 0x7f96ac140660 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.590 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1527526663 0 0) 0x7f96ac140140 con 0x7f96ac140f80 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9690004eb0 con 0x7f96ac140f80 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96a4001b40 con 0x7f96ac140f80 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.591 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687586654 0 0) 0x7f96ac1423a0 con 0x7f96ac09ac40 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ac140140 con 0x7f96ac09ac40 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96a00018b0 con 0x7f96ac09ac40 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.592 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 487838416 0 0) 0x7f96ac0b8040 con 0x7f96ac140660 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 >> v1:172.21.15.138:6790/0 conn(0x7f96ac09ac40 legacy=0x7f96ac1d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 >> v1:172.21.15.138:6789/0 conn(0x7f96ac140f80 legacy=0x7f96ac1e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 monclient: found mon.b 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac0b81d0 con 0x7f96ac140660 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 10 monclient: _renew_subs 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac1d7b90 con 0x7f96ac140660 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 10 monclient: _renew_subs 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9698004bf0 con 0x7f96ac140660 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac0b19f0 con 0x7f96ac140660 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96b3154640 1 librados: init done 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9698004f00 con 0x7f96ac140660 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:56.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 monclient: dump: 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f96980053f0 con 0x7f96ac140660 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96aaffd640 4 set_mon_vals no callback set 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2877565703 0 0) 0x7f9698006970 con 0x7f96ac140660 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 10 client.4105.objecter ms_dispatch 0x7f96ac000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 3 client.4105.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:56.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 3 client.4105.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.597+0000 7f96a97fa640 20 client.4105.objecter dump_active .. 0 homeless 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.598+0000 7f96b3154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.598+0000 7f96b3154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.598+0000 7f96b3154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.598+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac13ffa0 con 0x7f96ac140660 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.600+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f96ac13ffa0 con 0x7f96ac140660 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.600+0000 7f96a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.600+0000 7f96a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:56.650 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.672+0000 7f96b3154640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1d49822d-afc6-4c38-9d56-52b09808838f", "id": 0}] 2024-03-29T11:19:56.650 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.672+0000 7f96b3154640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1d49822d-afc6-4c38-9d56-52b09808838f", "id": 0}] 2024-03-29T11:19:56.650 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.672+0000 7f96b3154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.650 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.672+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1d49822d-afc6-4c38-9d56-52b09808838f", "id": 0} v 0) v1 -- 0x7f96ac1421c0 con 0x7f96ac140660 2024-03-29T11:19:56.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.675+0000 7f96a97fa640 1 -- 172.21.15.152:0/2788584250 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1d49822d-afc6-4c38-9d56-52b09808838f", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1164997366 0 732707515) 0x7f9698006b30 con 0x7f96ac140660 2024-03-29T11:19:56.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.675+0000 7f96a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1d49822d-afc6-4c38-9d56-52b09808838f", "id": 0}] 2024-03-29T11:19:56.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.675+0000 7f96a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:56.653 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 10 librados: watch_flush enter 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 10 librados: watch_flush exit 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 20 client.4105.objecter shutdown clearing up homeless session... 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 10 client.4105.objecter successfully canceled tick 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 10 monclient: shutdown 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 >> v1:172.21.15.152:6789/0 conn(0x7f96ac140660 legacy=0x7f96ac0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 shutdown_connections 2024-03-29T11:19:56.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 >> 172.21.15.152:0/2788584250 conn(0x7f96ac066b00 msgr2=0x7f96ac13c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 shutdown_connections 2024-03-29T11:19:56.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 1 -- 172.21.15.152:0/2788584250 wait complete. 2024-03-29T11:19:56.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.676+0000 7f96b3154640 1 librados: shutdown 2024-03-29T11:19:56.664 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 4ec3ef4f-0970-4f8e-ad05-505a67a88dbd 1 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.786+0000 7f6accba8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.786+0000 7f6accba8640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.786+0000 7f6accba8640 10 monclient: monmap: 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:56.788130+0000 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:56.788130+0000 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.764 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.765 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:56.765 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:56.765 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:56.765 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.787+0000 7f6accba8640 1 Processor -- start 2024-03-29T11:19:56.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.787+0000 7f6accba8640 1 -- start start 2024-03-29T11:19:56.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.787+0000 7f6accba8640 10 monclient: init 2024-03-29T11:19:56.768 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.787+0000 7f6accba8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.768 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.787+0000 7f6accba8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:56.768 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.787+0000 7f6accba8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac809ac40 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.787+0000 7f6accba8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac809c870 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac80b4360 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac809dec0 con 0x7f6ac80b4360 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac809da30 con 0x7f6ac809ac40 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac80b39c0 con 0x7f6ac809c870 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 10 monclient(hunting): authenticate will time out at 817.084956s 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac659c640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f6ac809ac40 0x7f6ac80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37364/0 (socket says 172.21.15.152:37364) 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac659c640 1 -- 172.21.15.152:0/3267475801 learned_addr learned my addr 172.21.15.152:0/3267475801 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578380877 0 0) 0x7f6ac809da30 con 0x7f6ac809ac40 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab0003520 con 0x7f6ac809ac40 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903287905 0 0) 0x7f6ab0003520 con 0x7f6ac809ac40 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 >> v1:172.21.15.138:6790/0 conn(0x7f6ac809c870 legacy=0x7f6ac813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 >> v1:172.21.15.138:6789/0 conn(0x7f6ac80b4360 legacy=0x7f6ac814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 monclient: found mon.noname-c 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac80b5160 con 0x7f6ac809ac40 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6ab8001690 con 0x7f6ac809ac40 2024-03-29T11:19:56.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: dump: 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.771 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ab8001870 con 0x7f6ac809ac40 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 1 -- 172.21.15.152:0/3267475801 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6ab8001c80 con 0x7f6ac809ac40 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6ac559a640 10 monclient: dump: 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.772 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.788+0000 7f6accba8640 4 set_mon_vals no callback set 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.789+0000 7f6accba8640 10 monclient: shutdown 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.789+0000 7f6accba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.789+0000 7f6accba8640 1 -- 172.21.15.152:0/3267475801 >> v1:172.21.15.152:6789/0 conn(0x7f6ac809ac40 legacy=0x7f6ac80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.789+0000 7f6accba8640 1 -- 172.21.15.152:0/3267475801 shutdown_connections 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.789+0000 7f6accba8640 1 -- 172.21.15.152:0/3267475801 >> 172.21.15.152:0/3267475801 conn(0x7f6ac8066b00 msgr2=0x7f6ac8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.789+0000 7f6accba8640 1 -- 172.21.15.152:0/3267475801 shutdown_connections 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.789+0000 7f6accba8640 1 -- 172.21.15.152:0/3267475801 wait complete. 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.790+0000 7f6accba8640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.790+0000 7f6accba8640 10 monclient: monmap: 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:56.773 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.791+0000 7f6accba8640 1 librados: starting msgr at 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.791+0000 7f6accba8640 1 librados: starting objecter 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.791+0000 7f6accba8640 1 Processor -- start 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.791+0000 7f6accba8640 1 -- start start 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.791+0000 7f6accba8640 1 librados: setting wanted keys 2024-03-29T11:19:56.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.791+0000 7f6accba8640 1 librados: calling monclient init 2024-03-29T11:19:56.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.791+0000 7f6accba8640 10 monclient: init 2024-03-29T11:19:56.780 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.780 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac809ac40 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac80b0f60 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac81cb0d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac813a7f0 con 0x7f6ac809ac40 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac8141c10 con 0x7f6ac80b0f60 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac8141ef0 con 0x7f6ac81cb0d0 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6accba8640 10 monclient(hunting): authenticate will time out at 817.089153s 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ac5d9b640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f6ac80b0f60 0x7f6ac81399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37370/0 (socket says 172.21.15.152:37370) 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ac5d9b640 1 -- 172.21.15.152:0/192934263 learned_addr learned my addr 172.21.15.152:0/192934263 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ac80b0f60 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ac809ac40 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ac81cb0d0 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873261982 0 0) 0x7f6ac8141c10 con 0x7f6ac80b0f60 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa8004eb0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726473809 0 0) 0x7f6ac8141ef0 con 0x7f6ac81cb0d0 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac8141c10 con 0x7f6ac81cb0d0 2024-03-29T11:19:56.782 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651103554 0 0) 0x7f6ac813a7f0 con 0x7f6ac809ac40 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac8141ef0 con 0x7f6ac809ac40 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3276043950 0 0) 0x7f6aa8004eb0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ac813a7f0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6abc0019d0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.783 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665669988 0 0) 0x7f6ac8141c10 con 0x7f6ac81cb0d0 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6aa8004eb0 con 0x7f6ac81cb0d0 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6ac0001970 con 0x7f6ac81cb0d0 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.784 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3162464231 0 0) 0x7f6ac8141ef0 con 0x7f6ac809ac40 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ac8141c10 con 0x7f6ac809ac40 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6ab8001b80 con 0x7f6ac809ac40 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:56.785 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.786 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.792+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3781110232 0 0) 0x7f6ac813a7f0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 >> v1:172.21.15.138:6790/0 conn(0x7f6ac81cb0d0 legacy=0x7f6ac813a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 >> v1:172.21.15.138:6789/0 conn(0x7f6ac809ac40 legacy=0x7f6ac813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 monclient: found mon.b 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac8142640 con 0x7f6ac80b0f60 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 10 monclient: _renew_subs 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac8142310 con 0x7f6ac80b0f60 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 10 monclient: _renew_subs 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac8142d40 con 0x7f6ac80b0f60 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 1 librados: init done 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6abc003610 con 0x7f6ac80b0f60 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6abc003a20 con 0x7f6ac80b0f60 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ac4d99640 4 set_mon_vals no callback set 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 monclient: dump: 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.791 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6abc003fd0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1550960025 0 0) 0x7f6abc0066f0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 10 client.4111.objecter ms_dispatch 0x7f6ac8000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 3 client.4111.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 3 client.4111.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6ab6ffd640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.793+0000 7f6accba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.794+0000 7f6accba8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.794+0000 7f6accba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.794+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aac005080 con 0x7f6ac80b0f60 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.796+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6abc006900 con 0x7f6ac80b0f60 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.796+0000 7f6ab6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.792 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.796+0000 7f6ab6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:56.845 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.868+0000 7f6accba8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4ec3ef4f-0970-4f8e-ad05-505a67a88dbd", "id": 1}] 2024-03-29T11:19:56.845 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.868+0000 7f6accba8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4ec3ef4f-0970-4f8e-ad05-505a67a88dbd", "id": 1}] 2024-03-29T11:19:56.845 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.868+0000 7f6accba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.845 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.868+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4ec3ef4f-0970-4f8e-ad05-505a67a88dbd", "id": 1} v 0) v1 -- 0x7f6aac0058b0 con 0x7f6ac80b0f60 2024-03-29T11:19:56.849 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.872+0000 7f6ab6ffd640 1 -- 172.21.15.152:0/192934263 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4ec3ef4f-0970-4f8e-ad05-505a67a88dbd", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2226142774 0 940483276) 0x7f6abc006b20 con 0x7f6ac80b0f60 2024-03-29T11:19:56.849 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.872+0000 7f6ab6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4ec3ef4f-0970-4f8e-ad05-505a67a88dbd", "id": 1}] 2024-03-29T11:19:56.849 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.872+0000 7f6ab6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:56.849 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 10 librados: watch_flush enter 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 10 librados: watch_flush exit 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 10 monclient: shutdown 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 >> v1:172.21.15.152:6789/0 conn(0x7f6ac80b0f60 legacy=0x7f6ac81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 shutdown_connections 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 >> 172.21.15.152:0/192934263 conn(0x7f6ac8066b00 msgr2=0x7f6ac813e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 shutdown_connections 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 1 -- 172.21.15.152:0/192934263 wait complete. 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.873+0000 7f6accba8640 1 librados: shutdown 2024-03-29T11:19:56.860 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 281d93c3-d8e5-4bf5-afeb-805e53e9d2be 2 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.985+0000 7fb906a9a640 10 monclient: get_monmap_and_config 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.985+0000 7fb906a9a640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.985+0000 7fb906a9a640 10 monclient: monmap: 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:56.986719+0000 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:56.986719+0000 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:56.963 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.972 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.985+0000 7fb906a9a640 1 Processor -- start 2024-03-29T11:19:56.972 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 1 -- start start 2024-03-29T11:19:56.972 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient: init 2024-03-29T11:19:56.972 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9000b1780 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient(hunting): picked mon.noname-b con 0x7fb90013d7e0 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient(hunting): picked mon.noname-a con 0x7fb900143be0 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb900070b30 con 0x7fb900143be0 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb900070da0 con 0x7fb9000b1780 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90009adb0 con 0x7fb90013d7e0 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb906a9a640 10 monclient(hunting): authenticate will time out at 817.283516s 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb904836640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fb9000b1780 0x7fb90013d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37380/0 (socket says 172.21.15.152:37380) 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb904836640 1 -- 172.21.15.152:0/1837897779 learned_addr learned my addr 172.21.15.152:0/1837897779 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734982502 0 0) 0x7fb900070da0 con 0x7fb9000b1780 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb8ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb8ff7fe640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.986+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f0003520 con 0x7fb9000b1780 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409157182 0 0) 0x7fb900070b30 con 0x7fb900143be0 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb900070da0 con 0x7fb900143be0 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798918613 0 0) 0x7fb90009adb0 con 0x7fb90013d7e0 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb900070b30 con 0x7fb90013d7e0 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635836067 0 0) 0x7fb8f0003520 con 0x7fb9000b1780 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 >> v1:172.21.15.138:6790/0 conn(0x7fb90013d7e0 legacy=0x7fb9001434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 >> v1:172.21.15.138:6789/0 conn(0x7fb900143be0 legacy=0x7fb90014b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb900144070 con 0x7fb9000b1780 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb8f4001690 con 0x7fb9000b1780 2024-03-29T11:19:56.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: dump: 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:19:56.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8f4001870 con 0x7fb9000b1780 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 1 -- 172.21.15.152:0/1837897779 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb8f4001c80 con 0x7fb9000b1780 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 4 set_mon_vals no callback set 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb8ff7fe640 10 monclient: dump: 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.977 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 10 monclient: shutdown 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 1 -- 172.21.15.152:0/1837897779 >> v1:172.21.15.152:6789/0 conn(0x7fb9000b1780 legacy=0x7fb90013d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 1 -- 172.21.15.152:0/1837897779 shutdown_connections 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 1 -- 172.21.15.152:0/1837897779 >> 172.21.15.152:0/1837897779 conn(0x7fb9000b58f0 msgr2=0x7fb9000b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.987+0000 7fb906a9a640 1 -- 172.21.15.152:0/1837897779 shutdown_connections 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.988+0000 7fb906a9a640 1 -- 172.21.15.152:0/1837897779 wait complete. 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.988+0000 7fb906a9a640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.988+0000 7fb906a9a640 10 monclient: monmap: 2024-03-29T11:19:56.978 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.988+0000 7fb906a9a640 1 librados: starting msgr at 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.988+0000 7fb906a9a640 1 librados: starting objecter 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.988+0000 7fb906a9a640 1 Processor -- start 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 1 -- start start 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 1 librados: setting wanted keys 2024-03-29T11:19:56.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 1 librados: calling monclient init 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient: init 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9000b1780 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient(hunting): picked mon.noname-c con 0x7fb90013d7e0 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient(hunting): picked mon.noname-b con 0x7fb900143be0 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb900135790 con 0x7fb9000b1780 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9001403e0 con 0x7fb900143be0 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9001406c0 con 0x7fb90013d7e0 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb905037640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fb900143be0 0x7fb900135080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37392/0 (socket says 172.21.15.152:37392) 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb904836640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fb9000b1780 0x7fb9001387c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.152:53414/0 (socket says 172.21.15.152:53414) 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb906a9a640 10 monclient(hunting): authenticate will time out at 817.286747s 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb905037640 1 -- 172.21.15.152:0/420939023 learned_addr learned my addr 172.21.15.152:0/420939023 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:56.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fb900143be0 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fb9000b1780 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fb90013d7e0 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.989+0000 7fb8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703848730 0 0) 0x7fb900135790 con 0x7fb9000b1780 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4004eb0 con 0x7fb9000b1780 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189274880 0 0) 0x7fb9001403e0 con 0x7fb900143be0 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb900135790 con 0x7fb900143be0 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668813552 0 0) 0x7fb9001406c0 con 0x7fb90013d7e0 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9001403e0 con 0x7fb90013d7e0 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2003802443 0 0) 0x7fb8e4004eb0 con 0x7fb9000b1780 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9001406c0 con 0x7fb9000b1780 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb8f4001b80 con 0x7fb9000b1780 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:56.982 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075260915 0 0) 0x7fb900135790 con 0x7fb900143be0 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e4004eb0 con 0x7fb900143be0 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3649714756 0 0) 0x7fb9001403e0 con 0x7fb90013d7e0 2024-03-29T11:19:56.983 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb900135790 con 0x7fb90013d7e0 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb8f8001b80 con 0x7fb900143be0 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.984 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb8ec0018f0 con 0x7fb90013d7e0 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3375601488 0 0) 0x7fb9001406c0 con 0x7fb9000b1780 2024-03-29T11:19:56.985 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 >> v1:172.21.15.138:6790/0 conn(0x7fb90013d7e0 legacy=0x7fb900134970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 >> v1:172.21.15.152:6789/0 conn(0x7fb900143be0 legacy=0x7fb900135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 monclient: found mon.a 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9001409a0 con 0x7fb9000b1780 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 5 monclient: authenticate success, global_id 4128 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 10 monclient: _renew_subs 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90013fa50 con 0x7fb9000b1780 2024-03-29T11:19:56.986 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 10 monclient: _renew_subs 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9001390b0 con 0x7fb9000b1780 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb906a9a640 1 librados: init done 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8f4004bf0 con 0x7fb9000b1780 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb8f4004f00 con 0x7fb9000b1780 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 monclient: dump: 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:56.987 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8feffd640 4 set_mon_vals no callback set 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb8f40053f0 con 0x7fb9000b1780 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2792070683 0 0) 0x7fb8f4005ac0 con 0x7fb9000b1780 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 10 client.4128.objecter ms_dispatch 0x7fb900000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.990+0000 7fb8fd7fa640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.991+0000 7fb906a9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.991+0000 7fb906a9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.991+0000 7fb906a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:56.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.991+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb900067c00 con 0x7fb9000b1780 2024-03-29T11:19:56.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.993+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb8f40056a0 con 0x7fb9000b1780 2024-03-29T11:19:56.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.993+0000 7fb8fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:56.993+0000 7fb8fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.007 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.031+0000 7fb653634040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:57.035 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.058+0000 7f7436c4d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:57.044 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.066+0000 7fb906a9a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "281d93c3-d8e5-4bf5-afeb-805e53e9d2be", "id": 2}] 2024-03-29T11:19:57.044 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.066+0000 7fb906a9a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "281d93c3-d8e5-4bf5-afeb-805e53e9d2be", "id": 2}] 2024-03-29T11:19:57.044 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.066+0000 7fb906a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:57.044 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.066+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd new", "uuid": "281d93c3-d8e5-4bf5-afeb-805e53e9d2be", "id": 2} v 0) v1 -- 0x7fb90004f730 con 0x7fb9000b1780 2024-03-29T11:19:57.045 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.068+0000 7fb8fd7fa640 1 -- 172.21.15.152:0/420939023 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "281d93c3-d8e5-4bf5-afeb-805e53e9d2be", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1247947938 0 216599125) 0x7fb8f4005d20 con 0x7fb9000b1780 2024-03-29T11:19:57.045 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.068+0000 7fb8fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "281d93c3-d8e5-4bf5-afeb-805e53e9d2be", "id": 2}] 2024-03-29T11:19:57.045 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.068+0000 7fb8fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.045 INFO:teuthology.orchestra.run.smithi152.stdout:2 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 10 librados: watch_flush enter 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 10 librados: watch_flush exit 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 10 client.4128.objecter successfully canceled tick 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 10 monclient: shutdown 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 >> v1:172.21.15.138:6789/0 conn(0x7fb9000b1780 legacy=0x7fb9001387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.047 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 shutdown_connections 2024-03-29T11:19:57.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 >> 172.21.15.152:0/420939023 conn(0x7fb9000b58f0 msgr2=0x7fb900069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.069+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 shutdown_connections 2024-03-29T11:19:57.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.070+0000 7fb906a9a640 1 -- 172.21.15.152:0/420939023 wait complete. 2024-03-29T11:19:57.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.070+0000 7fb906a9a640 1 librados: shutdown 2024-03-29T11:19:57.057 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 4defe436-16df-4982-aedd-de68b04aeb07 3 2024-03-29T11:19:57.061 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.086+0000 7fb653634040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:57.094 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.117+0000 7f7436c4d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:57.106 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.130+0000 7fb653634040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:57.142 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.164+0000 7f7436c4d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:57.157 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.180+0000 7fc062396640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.180+0000 7fc062396640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.180+0000 7fc062396640 10 monclient: monmap: 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:57.182353+0000 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:57.182353+0000 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:57.158 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.159 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.183+0000 7fb653634040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.181+0000 7fc062396640 1 Processor -- start 2024-03-29T11:19:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.181+0000 7fc062396640 1 -- start start 2024-03-29T11:19:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.181+0000 7fc062396640 10 monclient: init 2024-03-29T11:19:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 10 monclient(hunting): picked mon.noname-a con 0x7fc05c09ac40 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 10 monclient(hunting): picked mon.noname-b con 0x7fc05c09c870 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 10 monclient(hunting): picked mon.noname-c con 0x7fc05c0b4360 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c09dec0 con 0x7fc05c09ac40 2024-03-29T11:19:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c09da30 con 0x7fc05c0b4360 2024-03-29T11:19:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c0b39c0 con 0x7fc05c09c870 2024-03-29T11:19:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc062396640 10 monclient(hunting): authenticate will time out at 817.479289s 2024-03-29T11:19:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc060933640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fc05c0b4360 0x7fc05c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37402/0 (socket says 172.21.15.152:37402) 2024-03-29T11:19:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc060933640 1 -- 172.21.15.152:0/3286235419 learned_addr learned my addr 172.21.15.152:0/3286235419 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232626966 0 0) 0x7fc05c09da30 con 0x7fc05c0b4360 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc044003520 con 0x7fc05c0b4360 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721961055 0 0) 0x7fc050001110 con 0x7fc05c09ac40 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c09da30 con 0x7fc05c09ac40 2024-03-29T11:19:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764055063 0 0) 0x7fc05c0b39c0 con 0x7fc05c09c870 2024-03-29T11:19:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.174 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.175 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:57.175 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050001110 con 0x7fc05c09c870 2024-03-29T11:19:57.175 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2792500146 0 0) 0x7fc044003520 con 0x7fc05c0b4360 2024-03-29T11:19:57.175 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 >> v1:172.21.15.138:6790/0 conn(0x7fc05c09c870 legacy=0x7fc05c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.175 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 >> v1:172.21.15.138:6789/0 conn(0x7fc05c09ac40 legacy=0x7fc05c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.175 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.176 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.182+0000 7fc05affd640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.176 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.176 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.176 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c0b5160 con 0x7fc05c0b4360 2024-03-29T11:19:57.176 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.176 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc054001a10 con 0x7fc05c0b4360 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: dump: 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc054001b70 con 0x7fc05c0b4360 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 1 -- 172.21.15.152:0/3286235419 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc054001f80 con 0x7fc05c0b4360 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 4 set_mon_vals no callback set 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.178 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc05affd640 10 monclient: dump: 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 10 monclient: shutdown 2024-03-29T11:19:57.179 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 1 -- 172.21.15.152:0/3286235419 >> v1:172.21.15.152:6789/0 conn(0x7fc05c0b4360 legacy=0x7fc05c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 1 -- 172.21.15.152:0/3286235419 shutdown_connections 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 1 -- 172.21.15.152:0/3286235419 >> 172.21.15.152:0/3286235419 conn(0x7fc05c066b00 msgr2=0x7fc05c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 1 -- 172.21.15.152:0/3286235419 shutdown_connections 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.183+0000 7fc062396640 1 -- 172.21.15.152:0/3286235419 wait complete. 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 10 monclient: monmap: 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.180 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 1 librados: starting msgr at 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 1 librados: starting objecter 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 1 Processor -- start 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 1 -- start start 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 1 librados: setting wanted keys 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 1 librados: calling monclient init 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.184+0000 7fc062396640 10 monclient: init 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 10 monclient(hunting): picked mon.noname-c con 0x7fc05c09ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 10 monclient(hunting): picked mon.noname-a con 0x7fc05c140cc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 10 monclient(hunting): picked mon.noname-b con 0x7fc05c139420 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c141cf0 con 0x7fc05c140cc0 2024-03-29T11:19:57.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c0afb40 con 0x7fc05c139420 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c139890 con 0x7fc05c09ac40 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc062396640 10 monclient(hunting): authenticate will time out at 817.482423s 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc060933640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fc05c139420 0x7fc05c0affd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37416/0 (socket says 172.21.15.152:37416) 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc060933640 1 -- 172.21.15.152:0/532887730 learned_addr learned my addr 172.21.15.152:0/532887730 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 client.?.objecter ms_handle_connect 0x7fc05c139420 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 client.?.objecter ms_handle_connect 0x7fc05c09ac40 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 client.?.objecter ms_handle_connect 0x7fc05c140cc0 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174746169 0 0) 0x7fc05c0afb40 con 0x7fc05c139420 2024-03-29T11:19:57.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc048004eb0 con 0x7fc05c139420 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394780965 0 0) 0x7fc05c139890 con 0x7fc05c09ac40 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c0afb40 con 0x7fc05c09ac40 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931743343 0 0) 0x7fc05c141cf0 con 0x7fc05c140cc0 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:57.183 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c139890 con 0x7fc05c140cc0 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.185+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2019879626 0 0) 0x7fc048004eb0 con 0x7fc05c139420 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc05c141cf0 con 0x7fc05c139420 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc054001f40 con 0x7fc05c139420 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.184 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222657422 0 0) 0x7fc05c0afb40 con 0x7fc05c09ac40 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc048004eb0 con 0x7fc05c09ac40 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc0500018e0 con 0x7fc05c09ac40 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.185 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1460440272 0 0) 0x7fc05c139890 con 0x7fc05c140cc0 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc05c0afb40 con 0x7fc05c140cc0 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc04c0017f0 con 0x7fc05c140cc0 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.186 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 615267615 0 0) 0x7fc05c141cf0 con 0x7fc05c139420 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 >> v1:172.21.15.138:6790/0 conn(0x7fc05c09ac40 legacy=0x7fc05c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 >> v1:172.21.15.138:6789/0 conn(0x7fc05c140cc0 legacy=0x7fc05c138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 monclient: found mon.b 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.187 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c142b90 con 0x7fc05c139420 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 10 monclient: _renew_subs 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05c0b06e0 con 0x7fc05c139420 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 10 monclient: _renew_subs 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05c0b0b70 con 0x7fc05c139420 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc062396640 1 librados: init done 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc054004e10 con 0x7fc05c139420 2024-03-29T11:19:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc054005120 con 0x7fc05c139420 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc05a7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 monclient: dump: 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.189 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc054005610 con 0x7fc05c139420 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3506174568 0 0) 0x7fc054005d50 con 0x7fc05c139420 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 10 client.4123.objecter ms_dispatch 0x7fc05c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 3 client.4123.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 3 client.4123.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.186+0000 7fc058ff9640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.187+0000 7fc062396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.187+0000 7fc062396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.187+0000 7fc062396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.187+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c005080 con 0x7fc05c139420 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.189+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc0540058c0 con 0x7fc05c139420 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.189+0000 7fc058ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.190 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.189+0000 7fc058ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.201 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.222+0000 7f7436c4d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:57.202 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.226+0000 7fb653634040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:57.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.261+0000 7fc062396640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4defe436-16df-4982-aedd-de68b04aeb07", "id": 3}] 2024-03-29T11:19:57.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.261+0000 7fc062396640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4defe436-16df-4982-aedd-de68b04aeb07", "id": 3}] 2024-03-29T11:19:57.238 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.261+0000 7fc062396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.239 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.261+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4defe436-16df-4982-aedd-de68b04aeb07", "id": 3} v 0) v1 -- 0x7fc03c0058b0 con 0x7fc05c139420 2024-03-29T11:19:57.242 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.265+0000 7fc058ff9640 1 -- 172.21.15.152:0/532887730 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4defe436-16df-4982-aedd-de68b04aeb07", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3492054399 0 525046306) 0x7fc054005ae0 con 0x7fc05c139420 2024-03-29T11:19:57.242 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.265+0000 7fc058ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4defe436-16df-4982-aedd-de68b04aeb07", "id": 3}] 2024-03-29T11:19:57.242 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.265+0000 7fc058ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.242 INFO:teuthology.orchestra.run.smithi152.stdout:3 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 10 librados: watch_flush enter 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 10 librados: watch_flush exit 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 10 monclient: shutdown 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 >> v1:172.21.15.152:6789/0 conn(0x7fc05c139420 legacy=0x7fc05c0affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 shutdown_connections 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 >> 172.21.15.152:0/532887730 conn(0x7fc05c066b00 msgr2=0x7fc05c13a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 shutdown_connections 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 1 -- 172.21.15.152:0/532887730 wait complete. 2024-03-29T11:19:57.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.266+0000 7fc062396640 1 librados: shutdown 2024-03-29T11:19:57.245 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.267+0000 7f7436c4d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:57.253 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 02d2a411-15db-4cda-9b3d-9d9176a7c4d1 4 2024-03-29T11:19:57.319 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.343+0000 7fb653634040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:57.367 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.390+0000 7f7436c4d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:57.387 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.412+0000 7fb653634040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.418+0000 7f5d8284a640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.418+0000 7f5d8284a640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.418+0000 7f5d8284a640 10 monclient: monmap: 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:57.419661+0000 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:57.419661+0000 2024-03-29T11:19:57.395 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.396 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.396 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.396 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:57.396 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:57.396 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.418+0000 7f5d8284a640 1 Processor -- start 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.418+0000 7f5d8284a640 1 -- start start 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.418+0000 7f5d8284a640 10 monclient: init 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 10 monclient(hunting): picked mon.noname-b con 0x7f5d7c09ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7c09c870 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7c0b4360 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.404 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c09dec0 con 0x7f5d7c0b4360 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c09da30 con 0x7f5d7c09c870 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0b39c0 con 0x7f5d7c09ac40 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d8284a640 10 monclient(hunting): authenticate will time out at 817.716715s 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d7b7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f5d7c09c870 0x7f5d7c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37424/0 (socket says 172.21.15.152:37424) 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.419+0000 7f5d7b7fe640 1 -- 172.21.15.152:0/654882850 learned_addr learned my addr 172.21.15.152:0/654882850 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526009308 0 0) 0x7f5d7c09da30 con 0x7f5d7c09c870 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:57.405 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d64003520 con 0x7f5d7c09c870 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702061012 0 0) 0x7f5d7c0b39c0 con 0x7f5d7c09ac40 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c09da30 con 0x7f5d7c09ac40 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584863960 0 0) 0x7f5d74001490 con 0x7f5d7c0b4360 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c0b39c0 con 0x7f5d7c0b4360 2024-03-29T11:19:57.406 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3881756611 0 0) 0x7f5d64003520 con 0x7f5d7c09c870 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 >> v1:172.21.15.138:6790/0 conn(0x7f5d7c09ac40 legacy=0x7f5d7c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 >> v1:172.21.15.138:6789/0 conn(0x7f5d7c0b4360 legacy=0x7f5d7c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c0b5160 con 0x7f5d7c09c870 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d8284a640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5d70001690 con 0x7f5d7c09c870 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d8284a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: dump: 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d8284a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.408 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d700017f0 con 0x7f5d7c09c870 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 1 -- 172.21.15.152:0/654882850 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5d70001c40 con 0x7f5d7c09c870 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d8284a640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d8284a640 4 set_mon_vals no callback set 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.420+0000 7f5d7affd640 10 monclient: dump: 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.409 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 10 monclient: shutdown 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 1 -- 172.21.15.152:0/654882850 >> v1:172.21.15.152:6789/0 conn(0x7f5d7c09c870 legacy=0x7f5d7c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 1 -- 172.21.15.152:0/654882850 shutdown_connections 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 1 -- 172.21.15.152:0/654882850 >> 172.21.15.152:0/654882850 conn(0x7f5d7c066b00 msgr2=0x7f5d7c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 1 -- 172.21.15.152:0/654882850 shutdown_connections 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 1 -- 172.21.15.152:0/654882850 wait complete. 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.421+0000 7f5d8284a640 10 monclient: monmap: 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:57.410 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 librados: starting msgr at 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 librados: starting objecter 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 Processor -- start 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 -- start start 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 librados: setting wanted keys 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 librados: calling monclient init 2024-03-29T11:19:57.411 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient: init 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7c09ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient(hunting): picked mon.noname-b con 0x7f5d7c139b60 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7c146080 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0b1690 con 0x7f5d7c146080 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0b18c0 con 0x7f5d7c139b60 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c1dad10 con 0x7f5d7c09ac40 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.422+0000 7f5d8284a640 10 monclient(hunting): authenticate will time out at 817.719840s 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d7b7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f5d7c139b60 0x7f5d7c145970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37434/0 (socket says 172.21.15.152:37434) 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d7b7fe640 1 -- 172.21.15.152:0/635176509 learned_addr learned my addr 172.21.15.152:0/635176509 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d7bfff640 1 --1- 172.21.15.152:0/635176509 >> v1:172.21.15.138:6790/0 conn(0x7f5d7c09ac40 0x7f5d7c1da410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.152:36822/0 (socket says 172.21.15.152:36822) 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 client.?.objecter ms_handle_connect 0x7f5d7c139b60 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.412 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 client.?.objecter ms_handle_connect 0x7f5d7c09ac40 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 client.?.objecter ms_handle_connect 0x7f5d7c146080 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175526268 0 0) 0x7f5d7c0b18c0 con 0x7f5d7c139b60 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d68004eb0 con 0x7f5d7c139b60 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415603730 0 0) 0x7f5d7c1dad10 con 0x7f5d7c09ac40 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c0b18c0 con 0x7f5d7c09ac40 2024-03-29T11:19:57.413 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933634937 0 0) 0x7f5d7c0b1690 con 0x7f5d7c146080 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c1dad10 con 0x7f5d7c146080 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1999134304 0 0) 0x7f5d68004eb0 con 0x7f5d7c139b60 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d7c0b1690 con 0x7f5d7c139b60 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5d70001c40 con 0x7f5d7c139b60 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.414 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1870041109 0 0) 0x7f5d7c1dad10 con 0x7f5d7c146080 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d68004eb0 con 0x7f5d7c146080 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2170206206 0 0) 0x7f5d7c0b18c0 con 0x7f5d7c09ac40 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d7c1dad10 con 0x7f5d7c09ac40 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5d740019b0 con 0x7f5d7c146080 2024-03-29T11:19:57.415 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5d6c0017e0 con 0x7f5d7c09ac40 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.416 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 958702169 0 0) 0x7f5d7c0b1690 con 0x7f5d7c139b60 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 >> v1:172.21.15.138:6790/0 conn(0x7f5d7c09ac40 legacy=0x7f5d7c1da410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.417 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 >> v1:172.21.15.138:6789/0 conn(0x7f5d7c146080 legacy=0x7f5d7c0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 monclient: found mon.b 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c1464b0 con 0x7f5d7c139b60 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 10 monclient: _renew_subs 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c0aff30 con 0x7f5d7c139b60 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 10 monclient: _renew_subs 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c0b0460 con 0x7f5d7c139b60 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d8284a640 1 librados: init done 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d70004a90 con 0x7f5d7c139b60 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5d70004de0 con 0x7f5d7c139b60 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 10 monclient: dump: 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.419 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.423+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5d70005390 con 0x7f5d7c139b60 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 294447756 0 0) 0x7f5d700064f0 con 0x7f5d7c139b60 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d78ff9640 10 client.4129.objecter ms_dispatch 0x7f5d7c000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d78ff9640 3 client.4129.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d78ff9640 3 client.4129.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d78ff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d8284a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d8284a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d8284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.424+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c005080 con 0x7f5d7c139b60 2024-03-29T11:19:57.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.426+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5d70006720 con 0x7f5d7c139b60 2024-03-29T11:19:57.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.426+0000 7f5d78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.426+0000 7f5d78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.434 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.459+0000 7fb653634040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:57.441 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.464+0000 7f7436c4d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:57.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.498+0000 7f5d8284a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "02d2a411-15db-4cda-9b3d-9d9176a7c4d1", "id": 4}] 2024-03-29T11:19:57.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.498+0000 7f5d8284a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "02d2a411-15db-4cda-9b3d-9d9176a7c4d1", "id": 4}] 2024-03-29T11:19:57.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.498+0000 7f5d8284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.498+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd new", "uuid": "02d2a411-15db-4cda-9b3d-9d9176a7c4d1", "id": 4} v 0) v1 -- 0x7f5d5c0058b0 con 0x7f5d7c139b60 2024-03-29T11:19:57.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.502+0000 7f5d78ff9640 1 -- 172.21.15.152:0/635176509 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "02d2a411-15db-4cda-9b3d-9d9176a7c4d1", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1873473471 0 1697012583) 0x7f5d70006940 con 0x7f5d7c139b60 2024-03-29T11:19:57.480 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.502+0000 7f5d78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "02d2a411-15db-4cda-9b3d-9d9176a7c4d1", "id": 4}] 2024-03-29T11:19:57.480 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.502+0000 7f5d78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.480 INFO:teuthology.orchestra.run.smithi152.stdout:4 2024-03-29T11:19:57.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 10 librados: watch_flush enter 2024-03-29T11:19:57.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 10 librados: watch_flush exit 2024-03-29T11:19:57.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:19:57.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 10 monclient: shutdown 2024-03-29T11:19:57.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 >> v1:172.21.15.152:6789/0 conn(0x7f5d7c139b60 legacy=0x7f5d7c145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 shutdown_connections 2024-03-29T11:19:57.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 >> 172.21.15.152:0/635176509 conn(0x7f5d7c066b00 msgr2=0x7f5d7c1cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 shutdown_connections 2024-03-29T11:19:57.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.503+0000 7f5d8284a640 1 -- 172.21.15.152:0/635176509 wait complete. 2024-03-29T11:19:57.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.504+0000 7f5d8284a640 1 librados: shutdown 2024-03-29T11:19:57.488 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.512+0000 7fb653634040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:57.490 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.513+0000 7f7436c4d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:57.491 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a 5 2024-03-29T11:19:57.536 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.560+0000 7fb653634040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:57.548 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.570+0000 7f7436c4d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:57.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.615+0000 7faa4de93640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.615+0000 7faa4de93640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.615+0000 7faa4de93640 10 monclient: monmap: 2024-03-29T11:19:57.592 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.592 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:57.616618+0000 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:57.616618+0000 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:57.593 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.607 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.615+0000 7faa4de93640 1 Processor -- start 2024-03-29T11:19:57.607 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.615+0000 7faa4de93640 1 -- start start 2024-03-29T11:19:57.607 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.615+0000 7faa4de93640 10 monclient: init 2024-03-29T11:19:57.607 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.607 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 10 monclient(hunting): picked mon.noname-a con 0x7faa4809ac40 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 10 monclient(hunting): picked mon.noname-b con 0x7faa4809c870 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 10 monclient(hunting): picked mon.noname-c con 0x7faa480b4360 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4809dec0 con 0x7faa4809ac40 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4809da30 con 0x7faa480b4360 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480b39c0 con 0x7faa4809c870 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa4de93640 10 monclient(hunting): authenticate will time out at 817.913447s 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa47fff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7faa480b4360 0x7faa4814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37448/0 (socket says 172.21.15.152:37448) 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa47fff640 1 -- 172.21.15.152:0/561081971 learned_addr learned my addr 172.21.15.152:0/561081971 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286808809 0 0) 0x7faa4809da30 con 0x7faa480b4360 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa30003520 con 0x7faa480b4360 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037287938 0 0) 0x7faa38001110 con 0x7faa4809ac40 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.616+0000 7faa467fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa4809da30 con 0x7faa4809ac40 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044312918 0 0) 0x7faa480b39c0 con 0x7faa4809c870 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa38001110 con 0x7faa4809c870 2024-03-29T11:19:57.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1827527472 0 0) 0x7faa30003520 con 0x7faa480b4360 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 >> v1:172.21.15.138:6790/0 conn(0x7faa4809c870 legacy=0x7faa4813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 >> v1:172.21.15.138:6789/0 conn(0x7faa4809ac40 legacy=0x7faa480b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480b5160 con 0x7faa480b4360 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa40001a10 con 0x7faa480b4360 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: dump: 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa40001c60 con 0x7faa480b4360 2024-03-29T11:19:57.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 4 set_mon_vals no callback set 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 1 -- 172.21.15.152:0/561081971 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa40001ff0 con 0x7faa480b4360 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa467fc640 10 monclient: dump: 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.612 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 10 monclient: shutdown 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 1 -- 172.21.15.152:0/561081971 >> v1:172.21.15.152:6789/0 conn(0x7faa480b4360 legacy=0x7faa4814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 1 -- 172.21.15.152:0/561081971 shutdown_connections 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.617+0000 7faa4de93640 1 -- 172.21.15.152:0/561081971 >> 172.21.15.152:0/561081971 conn(0x7faa48066b00 msgr2=0x7faa48066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.618+0000 7faa4de93640 1 -- 172.21.15.152:0/561081971 shutdown_connections 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.618+0000 7faa4de93640 1 -- 172.21.15.152:0/561081971 wait complete. 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.618+0000 7faa4de93640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.618+0000 7faa4de93640 10 monclient: monmap: 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.613 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.619+0000 7faa4de93640 1 librados: starting msgr at 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.619+0000 7faa4de93640 1 librados: starting objecter 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.619+0000 7faa4de93640 1 Processor -- start 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.619+0000 7faa4de93640 1 -- start start 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.619+0000 7faa4de93640 1 librados: setting wanted keys 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.619+0000 7faa4de93640 1 librados: calling monclient init 2024-03-29T11:19:57.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.619+0000 7faa4de93640 10 monclient: init 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 10 monclient(hunting): picked mon.noname-a con 0x7faa4809ac40 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 10 monclient(hunting): picked mon.noname-c con 0x7faa481450a0 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 10 monclient(hunting): picked mon.noname-b con 0x7faa48142260 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa48145c60 con 0x7faa4809ac40 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa48146180 con 0x7faa48142260 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa481d7820 con 0x7faa481450a0 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa4de93640 10 monclient(hunting): authenticate will time out at 817.917537s 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa47fff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7faa48142260 0x7faa481e2a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37454/0 (socket says 172.21.15.152:37454) 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa477fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7faa4809ac40 0x7faa481d7110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.152:53458/0 (socket says 172.21.15.152:53458) 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa47fff640 1 -- 172.21.15.152:0/4121506611 learned_addr learned my addr 172.21.15.152:0/4121506611 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 client.?.objecter ms_handle_connect 0x7faa48142260 2024-03-29T11:19:57.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 client.?.objecter ms_handle_connect 0x7faa4809ac40 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 client.?.objecter ms_handle_connect 0x7faa481450a0 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307289035 0 0) 0x7faa48146180 con 0x7faa48142260 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa34004eb0 con 0x7faa48142260 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539976146 0 0) 0x7faa48145c60 con 0x7faa4809ac40 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa48146180 con 0x7faa4809ac40 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238882202 0 0) 0x7faa481d7820 con 0x7faa481450a0 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.620+0000 7faa2ffff640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa48145c60 con 0x7faa481450a0 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1071519238 0 0) 0x7faa34004eb0 con 0x7faa48142260 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa481d7820 con 0x7faa48142260 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa40001f70 con 0x7faa48142260 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:57.617 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897244035 0 0) 0x7faa48146180 con 0x7faa4809ac40 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa34004eb0 con 0x7faa4809ac40 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa38001920 con 0x7faa4809ac40 2024-03-29T11:19:57.618 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 609981003 0 0) 0x7faa48145c60 con 0x7faa481450a0 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa48146180 con 0x7faa481450a0 2024-03-29T11:19:57.619 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa3c001830 con 0x7faa481450a0 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.620 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 777174393 0 0) 0x7faa481d7820 con 0x7faa48142260 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 >> v1:172.21.15.138:6790/0 conn(0x7faa481450a0 legacy=0x7faa48137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 >> v1:172.21.15.138:6789/0 conn(0x7faa4809ac40 legacy=0x7faa481d7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 monclient: found mon.b 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa481d79b0 con 0x7faa48142260 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 10 monclient: _renew_subs 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa481d7c60 con 0x7faa48142260 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 10 monclient: _renew_subs 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa481388b0 con 0x7faa48142260 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa4de93640 1 librados: init done 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa40001570 con 0x7faa48142260 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa40004e10 con 0x7faa48142260 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa45ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 monclient: dump: 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faa40005300 con 0x7faa48142260 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3824548151 0 0) 0x7faa40006a10 con 0x7faa48142260 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 10 client.4135.objecter ms_dispatch 0x7faa48000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 3 client.4135.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 3 client.4135.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.621+0000 7faa2ffff640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.622+0000 7faa4de93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.623 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.622+0000 7faa4de93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.624 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.622+0000 7faa4de93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.624 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.622+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa28005080 con 0x7faa48142260 2024-03-29T11:19:57.624 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:57.622+0000 7f7436c4d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:57.624 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.624+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faa400065a0 con 0x7faa48142260 2024-03-29T11:19:57.624 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.624+0000 7faa2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.625 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.624+0000 7faa2ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.673 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.696+0000 7faa4de93640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a", "id": 5}] 2024-03-29T11:19:57.674 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.696+0000 7faa4de93640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a", "id": 5}] 2024-03-29T11:19:57.674 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.696+0000 7faa4de93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.674 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.696+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a", "id": 5} v 0) v1 -- 0x7faa280058b0 con 0x7faa48142260 2024-03-29T11:19:57.677 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.700+0000 7faa2ffff640 1 -- 172.21.15.152:0/4121506611 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2589096593 0 1988412176) 0x7faa40005790 con 0x7faa48142260 2024-03-29T11:19:57.677 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.700+0000 7faa2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a", "id": 5}] 2024-03-29T11:19:57.678 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.700+0000 7faa2ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.678 INFO:teuthology.orchestra.run.smithi152.stdout:5 2024-03-29T11:19:57.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 10 librados: watch_flush enter 2024-03-29T11:19:57.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 10 librados: watch_flush exit 2024-03-29T11:19:57.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:19:57.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 10 monclient: shutdown 2024-03-29T11:19:57.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.680 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 >> v1:172.21.15.152:6789/0 conn(0x7faa48142260 legacy=0x7faa481e2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 shutdown_connections 2024-03-29T11:19:57.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 >> 172.21.15.152:0/4121506611 conn(0x7faa48066b00 msgr2=0x7faa4813a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 shutdown_connections 2024-03-29T11:19:57.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 1 -- 172.21.15.152:0/4121506611 wait complete. 2024-03-29T11:19:57.681 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.701+0000 7faa4de93640 1 librados: shutdown 2024-03-29T11:19:57.689 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 43d86222-0bbc-4ba2-aee1-5c47d9f26662 6 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.811+0000 7f4aff38d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.811+0000 7f4aff38d640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.811+0000 7f4aff38d640 10 monclient: monmap: 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:57.813101+0000 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:57.813101+0000 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.789 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:57.790 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:57.790 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.790 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.812+0000 7f4aff38d640 1 Processor -- start 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.812+0000 7f4aff38d640 1 -- start start 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.812+0000 7f4aff38d640 10 monclient: init 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.812+0000 7f4aff38d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 10 monclient(hunting): picked mon.noname-c con 0x7f4af809ac40 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 10 monclient(hunting): picked mon.noname-b con 0x7f4af809c870 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 10 monclient(hunting): picked mon.noname-a con 0x7f4af80b4360 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af809dec0 con 0x7f4af80b4360 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af809da30 con 0x7f4af809ac40 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af80b39c0 con 0x7f4af809c870 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 10 monclient(hunting): authenticate will time out at 818.110021s 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4afd129640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f4af809ac40 0x7f4af80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37468/0 (socket says 172.21.15.152:37468) 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4afd129640 1 -- 172.21.15.152:0/38315856 learned_addr learned my addr 172.21.15.152:0/38315856 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508198926 0 0) 0x7f4af809da30 con 0x7f4af809ac40 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae4003520 con 0x7f4af809ac40 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856044672 0 0) 0x7f4af80b39c0 con 0x7f4af809c870 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af809da30 con 0x7f4af809c870 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740955250 0 0) 0x7f4af0001490 con 0x7f4af80b4360 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af80b39c0 con 0x7f4af80b4360 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979549386 0 0) 0x7f4ae4003520 con 0x7f4af809ac40 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 >> v1:172.21.15.138:6790/0 conn(0x7f4af809c870 legacy=0x7f4af813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 >> v1:172.21.15.138:6789/0 conn(0x7f4af80b4360 legacy=0x7f4af814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af80b5160 con 0x7f4af809ac40 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4af4001690 con 0x7f4af809ac40 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: dump: 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.801 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4af4001870 con 0x7f4af809ac40 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 1 -- 172.21.15.152:0/38315856 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4af4001c80 con 0x7f4af809ac40 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aff38d640 4 set_mon_vals no callback set 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.802 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.813+0000 7f4aeffff640 10 monclient: dump: 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.814+0000 7f4aff38d640 10 monclient: shutdown 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.814+0000 7f4aff38d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.814+0000 7f4aff38d640 1 -- 172.21.15.152:0/38315856 >> v1:172.21.15.152:6789/0 conn(0x7f4af809ac40 legacy=0x7f4af80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.803 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.814+0000 7f4aff38d640 1 -- 172.21.15.152:0/38315856 shutdown_connections 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.814+0000 7f4aff38d640 1 -- 172.21.15.152:0/38315856 >> 172.21.15.152:0/38315856 conn(0x7f4af8066b00 msgr2=0x7f4af8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.814+0000 7f4aff38d640 1 -- 172.21.15.152:0/38315856 shutdown_connections 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.814+0000 7f4aff38d640 1 -- 172.21.15.152:0/38315856 wait complete. 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 10 monclient: monmap: 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:57.804 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 1 librados: starting msgr at 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 1 librados: starting objecter 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 1 Processor -- start 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 1 -- start start 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 1 librados: setting wanted keys 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 1 librados: calling monclient init 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.815+0000 7f4aff38d640 10 monclient: init 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient(hunting): picked mon.noname-c con 0x7f4af809ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient(hunting): picked mon.noname-a con 0x7f4af80acd00 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient(hunting): picked mon.noname-b con 0x7f4af80af4c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af80b12f0 con 0x7f4af80acd00 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af80b15d0 con 0x7f4af80af4c0 2024-03-29T11:19:57.805 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af81d70e0 con 0x7f4af809ac40 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient(hunting): authenticate will time out at 818.113045s 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4afd92a640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f4af80af4c0 0x7f4af8139020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37478/0 (socket says 172.21.15.152:37478) 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4afd92a640 1 -- 172.21.15.152:0/2614422295 learned_addr learned my addr 172.21.15.152:0/2614422295 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 client.?.objecter ms_handle_connect 0x7f4af80af4c0 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 client.?.objecter ms_handle_connect 0x7f4af80acd00 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 client.?.objecter ms_handle_connect 0x7f4af809ac40 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.806 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841799244 0 0) 0x7f4af80b15d0 con 0x7f4af80af4c0 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc004eb0 con 0x7f4af80af4c0 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797850937 0 0) 0x7f4af80b12f0 con 0x7f4af80acd00 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af80b15d0 con 0x7f4af80acd00 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700047958 0 0) 0x7f4af81d70e0 con 0x7f4af809ac40 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.807 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af80b12f0 con 0x7f4af809ac40 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505952373 0 0) 0x7f4adc004eb0 con 0x7f4af80af4c0 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af81d70e0 con 0x7f4af80af4c0 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4af0001bc0 con 0x7f4af80af4c0 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): dump: 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.808 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3333432027 0 0) 0x7f4af80b12f0 con 0x7f4af809ac40 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4adc004eb0 con 0x7f4af809ac40 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4af4001b80 con 0x7f4af809ac40 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): dump: 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.809 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 285684132 0 0) 0x7f4af80b15d0 con 0x7f4af80acd00 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af80b12f0 con 0x7f4af80acd00 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4ae80017e0 con 0x7f4af80acd00 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient(hunting): dump: 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.810 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2820071744 0 0) 0x7f4af81d70e0 con 0x7f4af80af4c0 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 >> v1:172.21.15.138:6790/0 conn(0x7f4af809ac40 legacy=0x7f4af80b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 >> v1:172.21.15.138:6789/0 conn(0x7f4af80acd00 legacy=0x7f4af81d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 monclient: found mon.b 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af81d7d10 con 0x7f4af80af4c0 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient: _renew_subs 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.816+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af8139730 con 0x7f4af80af4c0 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4af0001fd0 con 0x7f4af80af4c0 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4af0003b30 con 0x7f4af80af4c0 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 10 monclient: dump: 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4af00040e0 con 0x7f4af80af4c0 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aef7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aff38d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aff38d640 10 monclient: _renew_subs 2024-03-29T11:19:57.813 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aff38d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af81e3ac0 con 0x7f4af80af4c0 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aff38d640 1 librados: init done 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1011396069 0 0) 0x7f4af00045c0 con 0x7f4af80af4c0 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 10 client.4141.objecter ms_dispatch 0x7f4af8000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 3 client.4141.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 3 client.4141.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.817+0000 7f4aedffb640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.818+0000 7f4aff38d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.818+0000 7f4aff38d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.818+0000 7f4aff38d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.818+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af8139ad0 con 0x7f4af80af4c0 2024-03-29T11:19:57.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.821+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4af0004800 con 0x7f4af80af4c0 2024-03-29T11:19:57.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.821+0000 7f4aedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.821+0000 7f4aedffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.871 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.894+0000 7f4aff38d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "43d86222-0bbc-4ba2-aee1-5c47d9f26662", "id": 6}] 2024-03-29T11:19:57.871 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.894+0000 7f4aff38d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "43d86222-0bbc-4ba2-aee1-5c47d9f26662", "id": 6}] 2024-03-29T11:19:57.871 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.894+0000 7f4aff38d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.872 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.894+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd new", "uuid": "43d86222-0bbc-4ba2-aee1-5c47d9f26662", "id": 6} v 0) v1 -- 0x7f4af809b020 con 0x7f4af80af4c0 2024-03-29T11:19:57.876 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.899+0000 7f4aedffb640 1 -- 172.21.15.152:0/2614422295 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "43d86222-0bbc-4ba2-aee1-5c47d9f26662", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2803076853 0 1113811849) 0x7f4af0005b40 con 0x7f4af80af4c0 2024-03-29T11:19:57.876 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.899+0000 7f4aedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "43d86222-0bbc-4ba2-aee1-5c47d9f26662", "id": 6}] 2024-03-29T11:19:57.876 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.899+0000 7f4aedffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.876 INFO:teuthology.orchestra.run.smithi152.stdout:6 2024-03-29T11:19:57.877 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.899+0000 7f4aff38d640 10 librados: watch_flush enter 2024-03-29T11:19:57.877 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.899+0000 7f4aff38d640 10 librados: watch_flush exit 2024-03-29T11:19:57.877 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.899+0000 7f4aff38d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.877 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.899+0000 7f4aff38d640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:19:57.877 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 10 monclient: shutdown 2024-03-29T11:19:57.878 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.878 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 >> v1:172.21.15.152:6789/0 conn(0x7f4af80af4c0 legacy=0x7f4af8139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.878 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 shutdown_connections 2024-03-29T11:19:57.878 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 >> 172.21.15.152:0/2614422295 conn(0x7f4af8066b00 msgr2=0x7f4af813c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.878 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 shutdown_connections 2024-03-29T11:19:57.878 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 1 -- 172.21.15.152:0/2614422295 wait complete. 2024-03-29T11:19:57.878 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:57.900+0000 7f4aff38d640 1 librados: shutdown 2024-03-29T11:19:57.887 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 05f0b6ff-da6f-45dc-8820-500924b513a4 7 2024-03-29T11:19:57.916 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:57.941+0000 7fb653634040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:57.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.010+0000 7efda2167640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.010+0000 7efda2167640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.987 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.010+0000 7efda2167640 10 monclient: monmap: 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:58.011851+0000 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:58.011851+0000 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:57.988 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 1 Processor -- start 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 1 -- start start 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient: init 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient(hunting): picked mon.noname-c con 0x7efd9c09ac40 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient(hunting): picked mon.noname-a con 0x7efd9c09c870 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient(hunting): picked mon.noname-b con 0x7efd9c0b4360 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c09dec0 con 0x7efd9c09c870 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c09da30 con 0x7efd9c09ac40 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0b39c0 con 0x7efd9c0b4360 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efda2167640 10 monclient(hunting): authenticate will time out at 818.308721s 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efd9b7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7efd9c09ac40 0x7efd9c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37480/0 (socket says 172.21.15.152:37480) 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.011+0000 7efd9b7fe640 1 -- 172.21.15.152:0/3608119013 learned_addr learned my addr 172.21.15.152:0/3608119013 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803309869 0 0) 0x7efd9c09da30 con 0x7efd9c09ac40 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd88003520 con 0x7efd9c09ac40 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375301539 0 0) 0x7efd9c0b39c0 con 0x7efd9c0b4360 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:19:57.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c09da30 con 0x7efd9c0b4360 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089364135 0 0) 0x7efd84001110 con 0x7efd9c09c870 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c0b39c0 con 0x7efd9c09c870 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3541670306 0 0) 0x7efd88003520 con 0x7efd9c09ac40 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 >> v1:172.21.15.138:6790/0 conn(0x7efd9c0b4360 legacy=0x7efd9c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 >> v1:172.21.15.138:6789/0 conn(0x7efd9c09c870 legacy=0x7efd9c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:19:57.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c0b5160 con 0x7efd9c09ac40 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efda2167640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efd8c001690 con 0x7efd9c09ac40 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efda2167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: dump: 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:58.000 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efda2167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd8c001870 con 0x7efd9c09ac40 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 1 -- 172.21.15.152:0/3608119013 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efd8c001c80 con 0x7efd9c09ac40 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efd9a7fc640 10 monclient: dump: 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.001 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efda2167640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.012+0000 7efda2167640 4 set_mon_vals no callback set 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 10 monclient: shutdown 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 1 -- 172.21.15.152:0/3608119013 >> v1:172.21.15.152:6789/0 conn(0x7efd9c09ac40 legacy=0x7efd9c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 1 -- 172.21.15.152:0/3608119013 shutdown_connections 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 1 -- 172.21.15.152:0/3608119013 >> 172.21.15.152:0/3608119013 conn(0x7efd9c066b00 msgr2=0x7efd9c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 1 -- 172.21.15.152:0/3608119013 shutdown_connections 2024-03-29T11:19:58.002 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 1 -- 172.21.15.152:0/3608119013 wait complete. 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.013+0000 7efda2167640 10 monclient: monmap: 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 librados: starting msgr at 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 librados: starting objecter 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 Processor -- start 2024-03-29T11:19:58.003 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 -- start start 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 librados: setting wanted keys 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 librados: calling monclient init 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient: init 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient(hunting): picked mon.noname-b con 0x7efd9c09ac40 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient(hunting): picked mon.noname-a con 0x7efd9c0acd70 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient(hunting): picked mon.noname-c con 0x7efd9c139820 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0b1a60 con 0x7efd9c0acd70 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c1d73f0 con 0x7efd9c09ac40 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c1d8020 con 0x7efd9c139820 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.014+0000 7efda2167640 10 monclient(hunting): authenticate will time out at 818.311838s 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd9b7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7efd9c09ac40 0x7efd9c1d7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:37488/0 (socket says 172.21.15.152:37488) 2024-03-29T11:19:58.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd9b7fe640 1 -- 172.21.15.152:0/1674687338 learned_addr learned my addr 172.21.15.152:0/1674687338 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 client.?.objecter ms_handle_connect 0x7efd9c09ac40 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 client.?.objecter ms_handle_connect 0x7efd9c0acd70 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 client.?.objecter ms_handle_connect 0x7efd9c139820 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295087636 0 0) 0x7efd9c1d73f0 con 0x7efd9c09ac40 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c004eb0 con 0x7efd9c09ac40 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867502203 0 0) 0x7efd9c0b1a60 con 0x7efd9c0acd70 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c1d73f0 con 0x7efd9c0acd70 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546125259 0 0) 0x7efd9c1d8020 con 0x7efd9c139820 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c0b1a60 con 0x7efd9c139820 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 23062246 0 0) 0x7efd9c1d73f0 con 0x7efd9c0acd70 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd9c1d8020 con 0x7efd9c0acd70 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642178386 0 0) 0x7efd7c004eb0 con 0x7efd9c09ac40 2024-03-29T11:19:58.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd9c1d73f0 con 0x7efd9c09ac40 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efd84001860 con 0x7efd9c0acd70 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.007 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2557673891 0 0) 0x7efd9c0b1a60 con 0x7efd9c139820 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd7c004eb0 con 0x7efd9c139820 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efd8c001b80 con 0x7efd9c09ac40 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.008 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efd940019f0 con 0x7efd9c139820 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.009 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 523100308 0 0) 0x7efd9c1d8020 con 0x7efd9c0acd70 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 >> v1:172.21.15.138:6790/0 conn(0x7efd9c139820 legacy=0x7efd9c0b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 >> v1:172.21.15.152:6789/0 conn(0x7efd9c09ac40 legacy=0x7efd9c1d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 monclient: found mon.a 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c1d81b0 con 0x7efd9c0acd70 2024-03-29T11:19:58.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 5 monclient: authenticate success, global_id 4158 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 10 monclient: _renew_subs 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c0b07b0 con 0x7efd9c0acd70 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 10 monclient: _renew_subs 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c0b0ce0 con 0x7efd9c0acd70 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efda2167640 1 librados: init done 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd84001cb0 con 0x7efd9c0acd70 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efd840037b0 con 0x7efd9c0acd70 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.011 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd99ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 monclient: dump: 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efd84003ca0 con 0x7efd9c0acd70 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3889474378 0 0) 0x7efd84005540 con 0x7efd9c0acd70 2024-03-29T11:19:58.012 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 10 client.4158.objecter ms_dispatch 0x7efd9c000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.015+0000 7efd93fff640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.016+0000 7efd93fff640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.016+0000 7efda2167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.016+0000 7efda2167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.016+0000 7efda2167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.016+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd80005080 con 0x7efd9c0acd70 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.018+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efd84004ff0 con 0x7efd9c0acd70 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.018+0000 7efd93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.013 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.018+0000 7efd93fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.013 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.030+0000 7f7436c4d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:58.052 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.076+0000 7fb653634040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:58.068 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.091+0000 7efda2167640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "05f0b6ff-da6f-45dc-8820-500924b513a4", "id": 7}] 2024-03-29T11:19:58.068 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.091+0000 7efda2167640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "05f0b6ff-da6f-45dc-8820-500924b513a4", "id": 7}] 2024-03-29T11:19:58.068 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.091+0000 7efda2167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:58.068 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.091+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd new", "uuid": "05f0b6ff-da6f-45dc-8820-500924b513a4", "id": 7} v 0) v1 -- 0x7efd800058b0 con 0x7efd9c0acd70 2024-03-29T11:19:58.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.092+0000 7efd93fff640 1 -- 172.21.15.152:0/1674687338 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "05f0b6ff-da6f-45dc-8820-500924b513a4", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1401922250 0 1371665406) 0x7efd84005360 con 0x7efd9c0acd70 2024-03-29T11:19:58.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.092+0000 7efd93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "05f0b6ff-da6f-45dc-8820-500924b513a4", "id": 7}] 2024-03-29T11:19:58.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.092+0000 7efd93fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.070 INFO:teuthology.orchestra.run.smithi152.stdout:7 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 10 librados: watch_flush enter 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 10 librados: watch_flush exit 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 10 client.4158.objecter successfully canceled tick 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 10 monclient: shutdown 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 >> v1:172.21.15.138:6789/0 conn(0x7efd9c0acd70 legacy=0x7efd9c139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 shutdown_connections 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 >> 172.21.15.152:0/1674687338 conn(0x7efd9c066b00 msgr2=0x7efd9c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 shutdown_connections 2024-03-29T11:19:58.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 1 -- 172.21.15.152:0/1674687338 wait complete. 2024-03-29T11:19:58.072 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-29T11:19:58.093+0000 7efda2167640 1 librados: shutdown 2024-03-29T11:19:58.080 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:58.080 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:58.083 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:58.083 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:58.083 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:58.085 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:58.085 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:58.085 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:58.087 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:58.087 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:58.087 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:58.089 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:58.089 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:58.089 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:58.092 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:58.092 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:58.092 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:58.094 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:58.094 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:58.094 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:58.096 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:58.096 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:58.097 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:58.099 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:58.099 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:58.177 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.200+0000 7f7436c4d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:58.199 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.224+0000 7fb653634040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.246+0000 7f8cbdd54640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.247+0000 7f8cbdd54640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.247+0000 7f8cbdd54640 10 monclient: monmap: 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:58.247279+0000 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:58.247279+0000 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.223 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.224 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:58.224 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:58.224 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:58.224 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.247+0000 7f8cbdd54640 1 Processor -- start 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.247+0000 7f8cbdd54640 1 -- start start 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.247+0000 7f8cbdd54640 10 monclient: init 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb809ac40 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb809c870 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cb77fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f8cb809ac40 0x7f8cb80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:60258/0 (socket says 172.21.15.138:60258) 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cb77fe640 1 -- 172.21.15.138:0/1809834852 learned_addr learned my addr 172.21.15.138:0/1809834852 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb80b4360 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb809dec0 con 0x7f8cb809c870 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb809da30 con 0x7f8cb80b4360 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80b39c0 con 0x7f8cb809ac40 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.248+0000 7f8cbdd54640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cbdd54640 10 monclient(hunting): authenticate will time out at 819.557065s 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526965832 0 0) 0x7f8ca8001110 con 0x7f8cb809c870 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca0003520 con 0x7f8cb809c870 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028770465 0 0) 0x7f8cb80b39c0 con 0x7f8cb809ac40 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:19:58.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca8001110 con 0x7f8cb809ac40 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164852671 0 0) 0x7f8cb809da30 con 0x7f8cb80b4360 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient(hunting): my global_id is 4174 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb80b39c0 con 0x7f8cb80b4360 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1984196792 0 0) 0x7f8ca0003520 con 0x7f8cb809c870 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 >> v1:172.21.15.138:6790/0 conn(0x7f8cb809ac40 legacy=0x7f8cb80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 >> v1:172.21.15.152:6789/0 conn(0x7f8cb80b4360 legacy=0x7f8cb814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 monclient: found mon.noname-a 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80b5160 con 0x7f8cb809c870 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8ca8001680 con 0x7f8cb809c870 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cbdd54640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: dump: 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cbdd54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ca8001860 con 0x7f8cb809c870 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 1 -- 172.21.15.138:0/1809834852 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8ca8001cb0 con 0x7f8cb809c870 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cb67fc640 10 monclient: dump: 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.239 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cbdd54640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.249+0000 7f8cbdd54640 4 set_mon_vals no callback set 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.250+0000 7f8cbdd54640 10 monclient: shutdown 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.250+0000 7f8cbdd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.250+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 >> v1:172.21.15.138:6789/0 conn(0x7f8cb809c870 legacy=0x7f8cb813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.250+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 shutdown_connections 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.250+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 >> 172.21.15.138:0/1809834852 conn(0x7f8cb8066b00 msgr2=0x7f8cb8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.250+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 shutdown_connections 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.250+0000 7f8cbdd54640 1 -- 172.21.15.138:0/1809834852 wait complete. 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 10 monclient: monmap: 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 1 librados: starting msgr at 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 1 librados: starting objecter 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 1 Processor -- start 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 1 -- start start 2024-03-29T11:19:58.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 1 librados: setting wanted keys 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 1 librados: calling monclient init 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.251+0000 7f8cbdd54640 10 monclient: init 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb809ac40 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb81d7c10 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb80b1350 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb81d7780 con 0x7f8cb809ac40 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb81406b0 con 0x7f8cb81d7c10 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb8140840 con 0x7f8cb80b1350 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cbdd54640 10 monclient(hunting): authenticate will time out at 819.560502s 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cb7fff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f8cb80b1350 0x7f8cb81e1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:60264/0 (socket says 172.21.15.138:60264) 2024-03-29T11:19:58.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8cb7fff640 1 -- 172.21.15.138:0/4099990942 learned_addr learned my addr 172.21.15.138:0/4099990942 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 client.?.objecter ms_handle_connect 0x7f8cb809ac40 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 client.?.objecter ms_handle_connect 0x7f8cb80b1350 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 client.?.objecter ms_handle_connect 0x7f8cb81d7c10 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618390176 0 0) 0x7f8cb8140840 con 0x7f8cb80b1350 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca4004eb0 con 0x7f8cb80b1350 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778596572 0 0) 0x7f8cb81d7780 con 0x7f8cb809ac40 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb8140840 con 0x7f8cb809ac40 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186693180 0 0) 0x7f8cb81406b0 con 0x7f8cb81d7c10 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.252+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb81d7780 con 0x7f8cb81d7c10 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764963111 0 0) 0x7f8ca4004eb0 con 0x7f8cb80b1350 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb81406b0 con 0x7f8cb80b1350 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8cb0001c70 con 0x7f8cb80b1350 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864105121 0 0) 0x7f8cb8140840 con 0x7f8cb809ac40 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ca4004eb0 con 0x7f8cb809ac40 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653057765 0 0) 0x7f8cb81d7780 con 0x7f8cb81d7c10 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb8140840 con 0x7f8cb81d7c10 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8cac001250 con 0x7f8cb809ac40 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.246 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8ca8001b70 con 0x7f8cb81d7c10 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.247 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 77116447 0 0) 0x7f8cb81406b0 con 0x7f8cb80b1350 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 >> v1:172.21.15.152:6789/0 conn(0x7f8cb81d7c10 legacy=0x7f8cb813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 >> v1:172.21.15.138:6789/0 conn(0x7f8cb809ac40 legacy=0x7f8cb81d6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 monclient: found mon.c 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80b75c0 con 0x7f8cb80b1350 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 10 monclient: _renew_subs 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb80b77b0 con 0x7f8cb80b1350 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 10 monclient: _renew_subs 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb81e3280 con 0x7f8cb80b1350 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb0003990 con 0x7f8cb80b1350 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cbdd54640 1 librados: init done 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8cb0003d60 con 0x7f8cb80b1350 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 monclient: dump: 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8cb5ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8cb0004310 con 0x7f8cb80b1350 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f8cb0006ca0 con 0x7f8cb80b1350 2024-03-29T11:19:58.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 10 client.4190.objecter ms_dispatch 0x7f8cb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.253+0000 7f8c9ffff640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.254+0000 7f8cbdd54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.254+0000 7f8cbdd54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.254+0000 7f8cbdd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.254+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c98005080 con 0x7f8cb80b1350 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.256+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8cb00047a0 con 0x7f8cb80b1350 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.256+0000 7f8c9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.256+0000 7f8c9ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.252 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.272+0000 7fb653634040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:58.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.317+0000 7f8cbdd54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.317+0000 7f8cbdd54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.317+0000 7f8cbdd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:19:58.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.317+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c98005330 con 0x7f8cb80b1350 2024-03-29T11:19:58.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.318+0000 7f8c9ffff640 1 -- 172.21.15.138:0/4099990942 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f8cb0006680 con 0x7f8cb80b1350 2024-03-29T11:19:58.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.318+0000 7f8c9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.318+0000 7f8c9ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.294 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-29T11:19:58.294 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:58.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 10 librados: watch_flush enter 2024-03-29T11:19:58.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 10 librados: watch_flush exit 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 10 client.4190.objecter successfully canceled tick 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 10 monclient: shutdown 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 >> v1:172.21.15.138:6790/0 conn(0x7f8cb80b1350 legacy=0x7f8cb81e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 shutdown_connections 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 >> 172.21.15.138:0/4099990942 conn(0x7f8cb8066b00 msgr2=0x7f8cb813bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 shutdown_connections 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 1 -- 172.21.15.138:0/4099990942 wait complete. 2024-03-29T11:19:58.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-29T11:19:58.319+0000 7f8cbdd54640 1 librados: shutdown 2024-03-29T11:19:58.303 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.326+0000 7f7436c4d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:58.306 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:58.306 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:58.349 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.372+0000 7f7436c4d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:58.365 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.389+0000 7fb653634040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:58.461 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.485+0000 7fb653634040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:58.462 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.485+0000 7f7436c4d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:58.559 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.582+0000 7f7436c4d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:58.615 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.639+0000 7fb653634040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:58.656 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.680+0000 7fb653634040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:58.695 INFO:tasks.ceph.osd.7.smithi152.stderr:2024-03-29T11:19:58.718+0000 7f24fdc12640 -1 Falling back to public interface 2024-03-29T11:19:58.698 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-03-29T11:19:58.721+0000 7f671e6ac640 -1 Falling back to public interface 2024-03-29T11:19:58.699 INFO:tasks.ceph.osd.5.smithi152.stderr:2024-03-29T11:19:58.722+0000 7fcca2a0c640 -1 Falling back to public interface 2024-03-29T11:19:58.707 INFO:tasks.ceph.osd.4.smithi152.stderr:2024-03-29T11:19:58.730+0000 7fa10aef7640 -1 Falling back to public interface 2024-03-29T11:19:58.709 INFO:tasks.ceph.osd.3.smithi138.stderr:2024-03-29T11:19:58.733+0000 7f113fd73640 -1 Falling back to public interface 2024-03-29T11:19:58.714 INFO:tasks.ceph.osd.0.smithi138.stderr:2024-03-29T11:19:58.738+0000 7fe4ae7fd640 -1 Falling back to public interface 2024-03-29T11:19:58.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2024-03-29T11:19:58.739+0000 7fb5c140d640 -1 Falling back to public interface 2024-03-29T11:19:58.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2024-03-29T11:19:58.739+0000 7f234b0d3640 -1 Falling back to public interface 2024-03-29T11:19:58.719 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.742+0000 7f7436c4d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:58.762 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.785+0000 7f7436c4d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:58.815 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:58.839+0000 7fb653634040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:58.915 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:58.938+0000 7f7436c4d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:59.111 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:59.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.225+0000 7f4522356640 10 monclient: get_monmap_and_config 2024-03-29T11:19:59.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.225+0000 7f4522356640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.225+0000 7f4522356640 10 monclient: monmap: 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:59.225980+0000 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:59.225980+0000 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:19:59.202 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.226+0000 7f4522356640 1 Processor -- start 2024-03-29T11:19:59.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.226+0000 7f4522356640 1 -- start start 2024-03-29T11:19:59.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.226+0000 7f4522356640 10 monclient: init 2024-03-29T11:19:59.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:59.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient(hunting): picked mon.noname-a con 0x7f451c1457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient(hunting): picked mon.noname-c con 0x7f451c13f650 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient(hunting): picked mon.noname-b con 0x7f451c0a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0aaef0 con 0x7f451c1457f0 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c145bd0 con 0x7f451c13f650 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c145eb0 con 0x7f451c0a9d70 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient(hunting): authenticate will time out at 820.535136s 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f45208f3640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f451c0a9d70 0x7f451c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:60276/0 (socket says 172.21.15.138:60276) 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451bfff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f451c1457f0 0x7f451c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47866/0 (socket says 172.21.15.138:47866) 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f45208f3640 1 -- 172.21.15.138:0/1950909078 learned_addr learned my addr 172.21.15.138:0/1950909078 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715525313 0 0) 0x7f451c0aaef0 con 0x7f451c1457f0 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:59.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4504003520 con 0x7f451c1457f0 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988596792 0 0) 0x7f451c145eb0 con 0x7f451c0a9d70 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0aaef0 con 0x7f451c0a9d70 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217922541 0 0) 0x7f451c145bd0 con 0x7f451c13f650 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c145eb0 con 0x7f451c13f650 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255582316 0 0) 0x7f4504003520 con 0x7f451c1457f0 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 >> v1:172.21.15.138:6790/0 conn(0x7f451c0a9d70 legacy=0x7f451c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 >> v1:172.21.15.152:6789/0 conn(0x7f451c13f650 legacy=0x7f451c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 monclient: found mon.noname-a 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c146340 con 0x7f451c1457f0 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f45100015c0 con 0x7f451c1457f0 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: dump: 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.213 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4510001720 con 0x7f451c1457f0 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 1 -- 172.21.15.138:0/1950909078 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4510001b70 con 0x7f451c1457f0 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f4522356640 4 set_mon_vals no callback set 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.227+0000 7f451affd640 10 monclient: dump: 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:19:59.214 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.228+0000 7f4522356640 10 monclient: shutdown 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.228+0000 7f4522356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.228+0000 7f4522356640 1 -- 172.21.15.138:0/1950909078 >> v1:172.21.15.138:6789/0 conn(0x7f451c1457f0 legacy=0x7f451c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.228+0000 7f4522356640 1 -- 172.21.15.138:0/1950909078 shutdown_connections 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.228+0000 7f4522356640 1 -- 172.21.15.138:0/1950909078 >> 172.21.15.138:0/1950909078 conn(0x7f451c066a00 msgr2=0x7f451c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.228+0000 7f4522356640 1 -- 172.21.15.138:0/1950909078 shutdown_connections 2024-03-29T11:19:59.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.228+0000 7f4522356640 1 -- 172.21.15.138:0/1950909078 wait complete. 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 10 monclient: monmap: 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 1 librados: starting msgr at 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 1 librados: starting objecter 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 1 Processor -- start 2024-03-29T11:19:59.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 1 -- start start 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 1 librados: setting wanted keys 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 1 librados: calling monclient init 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.229+0000 7f4522356640 10 monclient: init 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 10 monclient(hunting): picked mon.noname-a con 0x7f451c0a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 10 monclient(hunting): picked mon.noname-c con 0x7f451c13f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 10 monclient(hunting): picked mon.noname-b con 0x7f451c1457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c1d70f0 con 0x7f451c0a9d70 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0b03d0 con 0x7f451c1457f0 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0af100 con 0x7f451c13f650 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4522356640 10 monclient(hunting): authenticate will time out at 820.538489s 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f451bfff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f451c0a9d70 0x7f451c0adf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47874/0 (socket says 172.21.15.138:47874) 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f451bfff640 1 -- 172.21.15.138:0/1979493052 learned_addr learned my addr 172.21.15.138:0/1979493052 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:19:59.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 client.?.objecter ms_handle_connect 0x7f451c0a9d70 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 client.?.objecter ms_handle_connect 0x7f451c13f650 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 client.?.objecter ms_handle_connect 0x7f451c1457f0 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339793131 0 0) 0x7f451c1d70f0 con 0x7f451c0a9d70 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4508004eb0 con 0x7f451c0a9d70 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756983633 0 0) 0x7f451c0af100 con 0x7f451c13f650 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c1d70f0 con 0x7f451c13f650 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891646067 0 0) 0x7f451c0b03d0 con 0x7f451c1457f0 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0af100 con 0x7f451c1457f0 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.230+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3501878689 0 0) 0x7f4508004eb0 con 0x7f451c0a9d70 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f451c0b03d0 con 0x7f451c0a9d70 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4510001460 con 0x7f451c0a9d70 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:19:59.219 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 979560013 0 0) 0x7f451c1d70f0 con 0x7f451c13f650 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4508004eb0 con 0x7f451c13f650 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1861245573 0 0) 0x7f451c0af100 con 0x7f451c1457f0 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f451c1d70f0 con 0x7f451c1457f0 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f450c001870 con 0x7f451c13f650 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4514001430 con 0x7f451c1457f0 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:19:59.221 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3464537095 0 0) 0x7f451c0b03d0 con 0x7f451c0a9d70 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 >> v1:172.21.15.138:6790/0 conn(0x7f451c13f650 legacy=0x7f451c1d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 >> v1:172.21.15.152:6789/0 conn(0x7f451c1457f0 legacy=0x7f451c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 monclient: found mon.a 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:59.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0aec70 con 0x7f451c0a9d70 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 10 monclient: _renew_subs 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c0aeec0 con 0x7f451c0a9d70 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 10 monclient: _renew_subs 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c1d7da0 con 0x7f451c0a9d70 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4522356640 1 librados: init done 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4510004a90 con 0x7f451c0a9d70 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4510001180 con 0x7f451c0a9d70 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 monclient: dump: 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f451a7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4510005090 con 0x7f451c0a9d70 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f45100062c0 con 0x7f451c0a9d70 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 10 client.4194.objecter ms_dispatch 0x7f451c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:59.224 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.231+0000 7f4518ff9640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.232+0000 7f4522356640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.232+0000 7f4522356640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.232+0000 7f4522356640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.232+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc005080 con 0x7f451c0a9d70 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.234+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4510006570 con 0x7f451c0a9d70 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.234+0000 7f4518ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.234+0000 7f4518ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:59.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.296+0000 7f4522356640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.296+0000 7f4522356640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.296+0000 7f4522356640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:19:59.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.296+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44fc005330 con 0x7f451c0a9d70 2024-03-29T11:19:59.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.296+0000 7f4518ff9640 1 -- 172.21.15.138:0/1979493052 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f45100067d0 con 0x7f451c0a9d70 2024-03-29T11:19:59.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.296+0000 7f4518ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.296+0000 7f4518ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:59.273 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:19:59.273 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:59.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 10 librados: watch_flush enter 2024-03-29T11:19:59.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 10 librados: watch_flush exit 2024-03-29T11:19:59.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-29T11:19:59.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 10 client.4194.objecter successfully canceled tick 2024-03-29T11:19:59.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 10 monclient: shutdown 2024-03-29T11:19:59.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 >> v1:172.21.15.138:6789/0 conn(0x7f451c0a9d70 legacy=0x7f451c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 shutdown_connections 2024-03-29T11:19:59.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 >> 172.21.15.138:0/1979493052 conn(0x7f451c066a00 msgr2=0x7f451c13cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 shutdown_connections 2024-03-29T11:19:59.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 1 -- 172.21.15.138:0/1979493052 wait complete. 2024-03-29T11:19:59.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:19:59.297+0000 7f4522356640 1 librados: shutdown 2024-03-29T11:19:59.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:59.740 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:19:59.763+0000 7f7436c4d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:59.906 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:19:59.930+0000 7fb653634040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:20:00.015 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:20:00.038+0000 7f7436c4d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:00.070 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:20:00.091+0000 7f7436c4d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:00.180 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:20:00.204+0000 7fb653634040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:00.183 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:20:00.206+0000 7f7436c4d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:00.237 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:20:00.261+0000 7fb653634040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:00.352 INFO:tasks.ceph.mgr.y.smithi138.stderr:2024-03-29T11:20:00.376+0000 7fb653634040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:01.198 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:20:01.221+0000 7f73d7837640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:20:01.199 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-03-29T11:20:01.222+0000 7f73d7837640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:20:02.161 INFO:tasks.ceph.osd.5.smithi152.stderr:2024-03-29T11:20:02.184+0000 7fcca2a0c640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:20:02.215 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-03-29T11:20:02.238+0000 7f671e6ac640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:20:02.218 INFO:tasks.ceph.osd.4.smithi152.stderr:2024-03-29T11:20:02.241+0000 7fa10aef7640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:20:02.220 INFO:tasks.ceph.osd.7.smithi152.stderr:2024-03-29T11:20:02.243+0000 7f24fdc12640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:20:02.249 INFO:tasks.ceph.osd.0.smithi138.stderr:2024-03-29T11:20:02.273+0000 7fe4ae7fd640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:20:02.255 INFO:tasks.ceph.osd.2.smithi138.stderr:2024-03-29T11:20:02.280+0000 7fb5c140d640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:20:02.256 INFO:tasks.ceph.osd.3.smithi138.stderr:2024-03-29T11:20:02.280+0000 7f113fd73640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:20:02.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2024-03-29T11:20:02.280+0000 7f234b0d3640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:20:06.089 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.264+0000 7f180759e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.264+0000 7f180759e640 10 monclient: build_initial_monmap 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.264+0000 7f180759e640 10 monclient: monmap: 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:06.265216+0000 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:06.265216+0000 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:06.241 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:06.242 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:06.242 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:06.242 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:06.242 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.265+0000 7f180759e640 1 Processor -- start 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.266+0000 7f180759e640 1 -- start start 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.266+0000 7f180759e640 10 monclient: init 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 10 monclient(hunting): picked mon.noname-a con 0x7f1800006a70 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180659c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f1800006a70 0x7f18000c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:53466/0 (socket says 172.21.15.138:53466) 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 10 monclient(hunting): picked mon.noname-b con 0x7f180002b100 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 10 monclient(hunting): picked mon.noname-c con 0x7f1800029c30 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180002a270 con 0x7f1800006a70 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180002c440 con 0x7f1800029c30 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180659c640 1 -- 172.21.15.138:0/469100939 learned_addr learned my addr 172.21.15.138:0/469100939 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:06.245 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 1 -- 172.21.15.138:0/469100939 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18000c9440 con 0x7f180002b100 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.267+0000 7f180759e640 10 monclient(hunting): authenticate will time out at 827.575365s 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558204715 0 0) 0x7f18000c9440 con 0x7f180002b100 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f4003520 con 0x7f180002b100 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911157640 0 0) 0x7f180002c440 con 0x7f1800029c30 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.246 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): my global_id is 4222 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18000c9440 con 0x7f1800029c30 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774104005 0 0) 0x7f180002a270 con 0x7f1800006a70 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f180002c440 con 0x7f1800006a70 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 427613542 0 0) 0x7f17f4003520 con 0x7f180002b100 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 >> v1:172.21.15.152:6789/0 conn(0x7f1800029c30 legacy=0x7f18000d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 >> v1:172.21.15.138:6789/0 conn(0x7f1800006a70 legacy=0x7f18000c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 monclient: found mon.noname-b 2024-03-29T11:20:06.247 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18000c95d0 con 0x7f180002b100 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient: _finish_auth 0 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f17f0001610 con 0x7f180002b100 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180559a640 10 monclient: dump: 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.248 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.249 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:06.249 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:06.249 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:06.249 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:06.249 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:06.249 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180759e640 5 monclient: authenticate success, global_id 4232 2024-03-29T11:20:06.249 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.268+0000 7f180759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17f0001770 con 0x7f180002b100 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180559a640 1 -- 172.21.15.138:0/469100939 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f17f0001bc0 con 0x7f180002b100 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180559a640 10 monclient: dump: 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:06.256 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.269+0000 7f180759e640 4 set_mon_vals no callback set 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.270+0000 7f180759e640 10 monclient: shutdown 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.270+0000 7f180759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:06.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.270+0000 7f180759e640 1 -- 172.21.15.138:0/469100939 >> v1:172.21.15.138:6790/0 conn(0x7f180002b100 legacy=0x7f18000cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.270+0000 7f180759e640 1 -- 172.21.15.138:0/469100939 shutdown_connections 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.270+0000 7f180759e640 1 -- 172.21.15.138:0/469100939 >> 172.21.15.138:0/469100939 conn(0x7f18000058f0 msgr2=0x7f1800005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.271+0000 7f180759e640 1 -- 172.21.15.138:0/469100939 shutdown_connections 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.271+0000 7f180759e640 1 -- 172.21.15.138:0/469100939 wait complete. 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 10 monclient: build_initial_monmap 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 10 monclient: monmap: 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:06.258 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 1 librados: starting msgr at 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 1 librados: starting objecter 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 1 Processor -- start 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 1 -- start start 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 1 librados: setting wanted keys 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 1 librados: calling monclient init 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.272+0000 7f180759e640 10 monclient: init 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.273+0000 7f180759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.273+0000 7f180759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.273+0000 7f180759e640 10 monclient(hunting): picked mon.noname-a con 0x7f1800006a70 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:06.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.273+0000 7f180759e640 10 monclient(hunting): picked mon.noname-b con 0x7f1800029c30 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.273+0000 7f180759e640 10 monclient(hunting): picked mon.noname-c con 0x7f1800155470 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.273+0000 7f180659c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f1800006a70 0x7f1800158260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:53592/0 (socket says 172.21.15.138:53592) 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.273+0000 7f180659c640 1 -- 172.21.15.138:0/1238267254 learned_addr learned my addr 172.21.15.138:0/1238267254 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f17feffd640 10 client.?.objecter ms_handle_connect 0x7f1800029c30 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f17feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f17feffd640 10 client.?.objecter ms_handle_connect 0x7f1800006a70 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f17feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1800158970 con 0x7f1800006a70 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f17feffd640 10 client.?.objecter ms_handle_connect 0x7f1800155470 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f17feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18001831a0 con 0x7f1800029c30 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18000329f0 con 0x7f1800155470 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f180759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.274+0000 7f180759e640 10 monclient(hunting): authenticate will time out at 827.582750s 2024-03-29T11:20:06.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480278689 0 0) 0x7f18001831a0 con 0x7f1800029c30 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): my global_id is 4225 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e8004eb0 con 0x7f1800029c30 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277872803 0 0) 0x7f17e8004eb0 con 0x7f1800029c30 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18001831a0 con 0x7f1800029c30 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f17f0001ac0 con 0x7f1800029c30 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): dump: 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:06.261 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645089345 0 0) 0x7f1800158970 con 0x7f1800006a70 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.275+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e8004eb0 con 0x7f1800006a70 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355716612 0 0) 0x7f18000329f0 con 0x7f1800155470 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1800158970 con 0x7f1800155470 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 237342724 0 0) 0x7f18001831a0 con 0x7f1800029c30 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 >> v1:172.21.15.138:6790/0 conn(0x7f1800155470 legacy=0x7f18000c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 >> v1:172.21.15.138:6789/0 conn(0x7f1800006a70 legacy=0x7f1800158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 1 monclient: found mon.b 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:06.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800183dd0 con 0x7f1800029c30 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f17feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 5 monclient: authenticate success, global_id 4225 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 10 monclient: _renew_subs 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18000c2d20 con 0x7f1800029c30 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 10 monclient: _renew_subs 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:06.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18000320c0 con 0x7f1800029c30 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.276+0000 7f180759e640 1 librados: init done 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.277+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17f0001c20 con 0x7f1800029c30 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.277+0000 7f17feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.277+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f17f0003a40 con 0x7f1800029c30 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.277+0000 7f17feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f180759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f17feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f17feffd640 10 monclient: dump: 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:06.265 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f180759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f180759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1800183330 con 0x7f1800029c30 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f17fffff640 4 set_mon_vals no callback set 2024-03-29T11:20:06.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.278+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f17f0003c60 con 0x7f1800029c30 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.279+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f17f00839b0 con 0x7f1800029c30 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.279+0000 7f17feffd640 10 client.4225.objecter ms_dispatch 0x7f1808000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.279+0000 7f17feffd640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.279+0000 7f17feffd640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.279+0000 7f17feffd640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.279+0000 7f180659c640 10 monclient: get_auth_request con 0x7f17e806dd80 auth_method 0 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.279+0000 7f17feffd640 10 client.4225.objecter ms_handle_connect 0x7f17e806dd80 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.286+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17f004f290 con 0x7f1800029c30 2024-03-29T11:20:06.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.286+0000 7f17feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.286+0000 7f17feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:06.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.653+0000 7f180759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.654+0000 7f180759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.654+0000 7f180759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:06.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.654+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1800004cd0 con 0x7f1800029c30 2024-03-29T11:20:06.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.655+0000 7f17feffd640 1 -- 172.21.15.138:0/1238267254 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f17f00511c0 con 0x7f1800029c30 2024-03-29T11:20:06.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.655+0000 7f17feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.655+0000 7f17feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:06.631 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:06.631 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:06.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 10 librados: watch_flush enter 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 10 librados: watch_flush exit 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 10 client.4225.objecter successfully canceled tick 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 >> v1:172.21.15.152:6816/3052585572 conn(0x7f17e806dd80 legacy=0x7f17e8070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 10 monclient: shutdown 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 >> v1:172.21.15.152:6789/0 conn(0x7f1800029c30 legacy=0x7f18001836c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 shutdown_connections 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 >> 172.21.15.138:0/1238267254 conn(0x7f18000058f0 msgr2=0x7f1800036d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 shutdown_connections 2024-03-29T11:20:06.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.659+0000 7f180759e640 1 -- 172.21.15.138:0/1238267254 wait complete. 2024-03-29T11:20:06.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:06.660+0000 7f180759e640 1 librados: shutdown 2024-03-29T11:20:06.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:13.456 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:13.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.627+0000 7f7f48faf640 10 monclient: get_monmap_and_config 2024-03-29T11:20:13.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.627+0000 7f7f48faf640 10 monclient: build_initial_monmap 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.627+0000 7f7f48faf640 10 monclient: monmap: 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:13.628922+0000 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:13.628922+0000 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:13.605 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:13.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.628+0000 7f7f48faf640 1 Processor -- start 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.628+0000 7f7f48faf640 1 -- start start 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.628+0000 7f7f48faf640 10 monclient: init 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 10 monclient(hunting): picked mon.noname-c con 0x7f7f441457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4413f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 10 monclient(hunting): picked mon.noname-a con 0x7f7f440a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f440aaef0 con 0x7f7f440a9d70 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f44145bd0 con 0x7f7f441457f0 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f44145eb0 con 0x7f7f4413f650 2024-03-29T11:20:13.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.629+0000 7f7f48faf640 10 monclient(hunting): authenticate will time out at 834.938924s 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f42d9d640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f7f440a9d70 0x7f7f4414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:43302/0 (socket says 172.21.15.138:43302) 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f42d9d640 1 -- 172.21.15.138:0/1330461961 learned_addr learned my addr 172.21.15.138:0/1330461961 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799736452 0 0) 0x7f7f44145bd0 con 0x7f7f441457f0 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c003520 con 0x7f7f441457f0 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908716860 0 0) 0x7f7f440aaef0 con 0x7f7f440a9d70 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f44145bd0 con 0x7f7f440a9d70 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193847647 0 0) 0x7f7f2c003520 con 0x7f7f441457f0 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 >> v1:172.21.15.138:6790/0 conn(0x7f7f4413f650 legacy=0x7f7f440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 >> v1:172.21.15.138:6789/0 conn(0x7f7f440a9d70 legacy=0x7f7f4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 monclient: found mon.noname-c 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.630+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f44146340 con 0x7f7f441457f0 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 10 monclient: _finish_auth 0 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f48faf640 5 monclient: authenticate success, global_id 4228 2024-03-29T11:20:13.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f48faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7f3c001a10 con 0x7f7f441457f0 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 10 monclient: dump: 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:13.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f48faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 1 -- 172.21.15.138:0/1330461961 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c001c30 con 0x7f7f441457f0 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f4159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f48faf640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.631+0000 7f7f48faf640 4 set_mon_vals no callback set 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.632+0000 7f7f48faf640 10 monclient: shutdown 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.632+0000 7f7f48faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.632+0000 7f7f48faf640 1 -- 172.21.15.138:0/1330461961 >> v1:172.21.15.152:6789/0 conn(0x7f7f441457f0 legacy=0x7f7f4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.632+0000 7f7f48faf640 1 -- 172.21.15.138:0/1330461961 shutdown_connections 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.632+0000 7f7f48faf640 1 -- 172.21.15.138:0/1330461961 >> 172.21.15.138:0/1330461961 conn(0x7f7f44066a00 msgr2=0x7f7f44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.632+0000 7f7f48faf640 1 -- 172.21.15.138:0/1330461961 shutdown_connections 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.632+0000 7f7f48faf640 1 -- 172.21.15.138:0/1330461961 wait complete. 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.633+0000 7f7f48faf640 10 monclient: build_initial_monmap 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.633+0000 7f7f48faf640 10 monclient: monmap: 2024-03-29T11:20:13.622 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.634+0000 7f7f48faf640 1 librados: starting msgr at 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.634+0000 7f7f48faf640 1 librados: starting objecter 2024-03-29T11:20:13.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.634+0000 7f7f48faf640 1 Processor -- start 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.634+0000 7f7f48faf640 1 -- start start 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.634+0000 7f7f48faf640 1 librados: setting wanted keys 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.634+0000 7f7f48faf640 1 librados: calling monclient init 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.634+0000 7f7f48faf640 10 monclient: init 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 10 monclient(hunting): picked mon.noname-c con 0x7f7f440a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4413f650 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 10 monclient(hunting): picked mon.noname-b con 0x7f7f441457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4406ed60 con 0x7f7f4413f650 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f44137790 con 0x7f7f441457f0 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f44142d10 con 0x7f7f440a9d70 2024-03-29T11:20:13.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f48faf640 10 monclient(hunting): authenticate will time out at 834.944611s 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f4259c640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f7f440a9d70 0x7f7f4413fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:34182/0 (socket says 172.21.15.138:34182) 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f4259c640 1 -- 172.21.15.138:0/2000626097 learned_addr learned my addr 172.21.15.138:0/2000626097 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f32ffd640 10 client.?.objecter ms_handle_connect 0x7f7f441457f0 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f32ffd640 10 client.?.objecter ms_handle_connect 0x7f7f440a9d70 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f32ffd640 10 client.?.objecter ms_handle_connect 0x7f7f4413f650 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.635+0000 7f7f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333557587 0 0) 0x7f7f44137790 con 0x7f7f441457f0 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:20:13.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f24004eb0 con 0x7f7f441457f0 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791329364 0 0) 0x7f7f4406ed60 con 0x7f7f4413f650 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f44137790 con 0x7f7f4413f650 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 235935717 0 0) 0x7f7f24004eb0 con 0x7f7f441457f0 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4406ed60 con 0x7f7f441457f0 2024-03-29T11:20:13.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7f340018b0 con 0x7f7f441457f0 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:13.627 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771195733 0 0) 0x7f7f44142d10 con 0x7f7f440a9d70 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.636+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f24004eb0 con 0x7f7f440a9d70 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3031371936 0 0) 0x7f7f4406ed60 con 0x7f7f441457f0 2024-03-29T11:20:13.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 >> v1:172.21.15.138:6790/0 conn(0x7f7f440a9d70 legacy=0x7f7f4413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 >> v1:172.21.15.138:6789/0 conn(0x7f7f4413f650 legacy=0x7f7f44136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 1 monclient: found mon.b 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f441427f0 con 0x7f7f441457f0 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 10 monclient: _renew_subs 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f441423a0 con 0x7f7f441457f0 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:13.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 10 monclient: _renew_subs 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f44144aa0 con 0x7f7f441457f0 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.637+0000 7f7f48faf640 1 librados: init done 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.638+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f34001cc0 con 0x7f7f441457f0 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.638+0000 7f7f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.638+0000 7f7f40d99640 4 set_mon_vals no callback set 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f48faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7f34004a40 con 0x7f7f441457f0 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f48faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f48faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28005080 con 0x7f7f441457f0 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.639+0000 7f7f32ffd640 10 monclient: dump: 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:13.631 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:13.632 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.641+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f7f34004ba0 con 0x7f7f441457f0 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.642+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f7f34084900 con 0x7f7f441457f0 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.642+0000 7f7f32ffd640 10 client.4231.objecter ms_dispatch 0x7f7f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.642+0000 7f7f32ffd640 3 client.4231.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.642+0000 7f7f32ffd640 3 client.4231.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.642+0000 7f7f32ffd640 20 client.4231.objecter dump_active .. 0 homeless 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.642+0000 7f7f4259c640 10 monclient: get_auth_request con 0x7f7f2406df20 auth_method 0 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.643+0000 7f7f32ffd640 10 client.4231.objecter ms_handle_connect 0x7f7f2406df20 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.648+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f340501e0 con 0x7f7f441457f0 2024-03-29T11:20:13.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.648+0000 7f7f32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.648+0000 7f7f32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:13.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.975+0000 7f7f48faf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.975+0000 7f7f48faf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.975+0000 7f7f48faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:13.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.975+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f28005330 con 0x7f7f441457f0 2024-03-29T11:20:13.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.976+0000 7f7f32ffd640 1 -- 172.21.15.138:0/2000626097 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f7f34052110 con 0x7f7f441457f0 2024-03-29T11:20:13.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.976+0000 7f7f32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.976+0000 7f7f32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:13.954 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:13.954 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 10 librados: watch_flush enter 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 10 librados: watch_flush exit 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 20 client.4231.objecter shutdown clearing up homeless session... 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 10 client.4231.objecter successfully canceled tick 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 >> v1:172.21.15.152:6816/3052585572 conn(0x7f7f2406df20 legacy=0x7f7f240703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 10 monclient: shutdown 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 >> v1:172.21.15.152:6789/0 conn(0x7f7f441457f0 legacy=0x7f7f44137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 shutdown_connections 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 >> 172.21.15.138:0/2000626097 conn(0x7f7f44066a00 msgr2=0x7f7f44139dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 shutdown_connections 2024-03-29T11:20:13.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.980+0000 7f7f48faf640 1 -- 172.21.15.138:0/2000626097 wait complete. 2024-03-29T11:20:13.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:13.981+0000 7f7f48faf640 1 librados: shutdown 2024-03-29T11:20:13.972 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:20.779 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.953+0000 7f8e21f97640 10 monclient: get_monmap_and_config 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.953+0000 7f8e21f97640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.953+0000 7f8e21f97640 10 monclient: monmap: 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:20.954529+0000 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:20.954529+0000 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:20.931 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:20.932 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:20.932 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.954+0000 7f8e21f97640 1 Processor -- start 2024-03-29T11:20:20.932 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.954+0000 7f8e21f97640 1 -- start start 2024-03-29T11:20:20.932 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.954+0000 7f8e21f97640 10 monclient: init 2024-03-29T11:20:20.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.955+0000 7f8e21f97640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.955+0000 7f8e21f97640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:20.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c0b0500 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:20.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c0b08e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:20.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c0aeca0 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:20.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c140620 con 0x7f8e1c0b08e0 2024-03-29T11:20:20.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c13daf0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c0ae180 con 0x7f8e1c0b0500 2024-03-29T11:20:20.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.956+0000 7f8e21f97640 10 monclient(hunting): authenticate will time out at 842.265167s 2024-03-29T11:20:20.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.957+0000 7f8e1b7fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f8e1c0b0500 0x7f8e1c13ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:54049/0 (socket says 172.21.15.138:54049) 2024-03-29T11:20:20.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.957+0000 7f8e1b7fe640 1 -- 172.21.15.138:0/509009297 learned_addr learned my addr 172.21.15.138:0/509009297 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311510624 0 0) 0x7f8e1c0ae180 con 0x7f8e1c0b0500 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e04003520 con 0x7f8e1c0b0500 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788395758 0 0) 0x7f8e04003520 con 0x7f8e1c0b0500 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 >> v1:172.21.15.152:6789/0 conn(0x7f8e1c0aeca0 legacy=0x7f8e1c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:20.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.958+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 >> v1:172.21.15.138:6789/0 conn(0x7f8e1c0b08e0 legacy=0x7f8e1c13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0accf0 con 0x7f8e1c0b0500 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.960+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8e10001690 con 0x7f8e1c0b0500 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 10 monclient: dump: 2024-03-29T11:20:20.938 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e21f97640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:20:20.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e21f97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:20.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e10001860 con 0x7f8e1c0b0500 2024-03-29T11:20:20.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 1 -- 172.21.15.138:0/509009297 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8e10001cb0 con 0x7f8e1c0b0500 2024-03-29T11:20:20.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:20.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e1a7fc640 10 monclient: dump: 2024-03-29T11:20:20.940 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e21f97640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.961+0000 7f8e21f97640 4 set_mon_vals no callback set 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.962+0000 7f8e21f97640 10 monclient: shutdown 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.962+0000 7f8e21f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:20.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.962+0000 7f8e21f97640 1 -- 172.21.15.138:0/509009297 >> v1:172.21.15.138:6790/0 conn(0x7f8e1c0b0500 legacy=0x7f8e1c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.962+0000 7f8e21f97640 1 -- 172.21.15.138:0/509009297 shutdown_connections 2024-03-29T11:20:20.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.962+0000 7f8e21f97640 1 -- 172.21.15.138:0/509009297 >> 172.21.15.138:0/509009297 conn(0x7f8e1c06b660 msgr2=0x7f8e1c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:20.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.962+0000 7f8e21f97640 1 -- 172.21.15.138:0/509009297 shutdown_connections 2024-03-29T11:20:20.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.963+0000 7f8e21f97640 1 -- 172.21.15.138:0/509009297 wait complete. 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.963+0000 7f8e21f97640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.963+0000 7f8e21f97640 10 monclient: monmap: 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:20.943 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.964+0000 7f8e21f97640 1 librados: starting msgr at 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.964+0000 7f8e21f97640 1 librados: starting objecter 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.964+0000 7f8e21f97640 1 Processor -- start 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.964+0000 7f8e21f97640 1 -- start start 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.964+0000 7f8e21f97640 1 librados: setting wanted keys 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.964+0000 7f8e21f97640 1 librados: calling monclient init 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.964+0000 7f8e21f97640 10 monclient: init 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.965+0000 7f8e21f97640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.966+0000 7f8e21f97640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:20.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.966+0000 7f8e21f97640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c0aeca0 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:20.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.968+0000 7f8e21f97640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c0b0500 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:20.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.968+0000 7f8e21f97640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c11ca80 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:20.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.968+0000 7f8e21f97640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c11e9b0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.969+0000 7f8e1b7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f8e1c0aeca0 0x7f8e1c1d4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:59037/0 (socket says 172.21.15.138:59037) 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.969+0000 7f8e1b7fe640 1 -- 172.21.15.138:0/604469087 learned_addr learned my addr 172.21.15.138:0/604469087 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.969+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c11ec00 con 0x7f8e1c0b0500 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c11d3c0 con 0x7f8e1c11ca80 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e21f97640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e21f97640 10 monclient(hunting): authenticate will time out at 842.279128s 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 10 client.?.objecter ms_handle_connect 0x7f8e1c0aeca0 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632852339 0 0) 0x7f8e1c11e9b0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:20:20.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.970+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e08004eb0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.972+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218277236 0 0) 0x7f8e08004eb0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.972+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e1c11e9b0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.972+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8e10001b70 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.972+0000 7f8e03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.972+0000 7f8e03fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 monclient(hunting): dump: 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 client.?.objecter ms_handle_connect 0x7f8e1c0b0500 2024-03-29T11:20:20.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063427904 0 0) 0x7f8e1c11e9b0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 >> v1:172.21.15.138:6790/0 conn(0x7f8e1c11ca80 legacy=0x7f8e1c1e06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 >> v1:172.21.15.152:6789/0 conn(0x7f8e1c0b0500 legacy=0x7f8e1c138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 1 monclient: found mon.a 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c11d070 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 10 monclient: _renew_subs 2024-03-29T11:20:20.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c1d5490 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 10 monclient: _renew_subs 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c11dac0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e21f97640 1 librados: init done 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e10004bf0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.973+0000 7f8e03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.974+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8e10004fc0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.974+0000 7f8e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.974+0000 7f8e19ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.974+0000 7f8e03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.974+0000 7f8e03fff640 10 monclient: dump: 2024-03-29T11:20:20.955 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.975+0000 7f8e21f97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.975+0000 7f8e21f97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.975+0000 7f8e21f97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:20:20.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.975+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc005080 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.976+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f8e10005120 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.977+0000 7f8e1affd640 10 monclient: get_auth_request con 0x7f8e0806dbb0 auth_method 0 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.977+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f8e10084ce0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.977+0000 7f8e03fff640 10 client.4251.objecter ms_dispatch 0x7f8e1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.977+0000 7f8e03fff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.977+0000 7f8e03fff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.977+0000 7f8e03fff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:20:20.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.977+0000 7f8e03fff640 10 client.4251.objecter ms_handle_connect 0x7f8e0806dbb0 2024-03-29T11:20:20.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.987+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e100505c0 con 0x7f8e1c0aeca0 2024-03-29T11:20:20.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.987+0000 7f8e03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:20.987+0000 7f8e03fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:21.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.317+0000 7f8e21f97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.317+0000 7f8e21f97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.317+0000 7f8e21f97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:20:21.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.317+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dfc005330 con 0x7f8e1c0aeca0 2024-03-29T11:20:21.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.318+0000 7f8e03fff640 1 -- 172.21.15.138:0/604469087 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f8e100524f0 con 0x7f8e1c0aeca0 2024-03-29T11:20:21.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.318+0000 7f8e03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.318+0000 7f8e03fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:21.296 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:21.296 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.321+0000 7f8e21f97640 10 librados: watch_flush enter 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.322+0000 7f8e21f97640 10 librados: watch_flush exit 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.322+0000 7f8e21f97640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.322+0000 7f8e21f97640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.322+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 >> v1:172.21.15.152:6816/3052585572 conn(0x7f8e0806dbb0 legacy=0x7f8e08070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.322+0000 7f8e21f97640 10 monclient: shutdown 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.322+0000 7f8e21f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:21.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.322+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 >> v1:172.21.15.138:6789/0 conn(0x7f8e1c0aeca0 legacy=0x7f8e1c1d4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.324+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 shutdown_connections 2024-03-29T11:20:21.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.324+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 >> 172.21.15.138:0/604469087 conn(0x7f8e1c06b660 msgr2=0x7f8e1c0a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:21.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.324+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 shutdown_connections 2024-03-29T11:20:21.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.324+0000 7f8e21f97640 1 -- 172.21.15.138:0/604469087 wait complete. 2024-03-29T11:20:21.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:21.325+0000 7f8e21f97640 1 librados: shutdown 2024-03-29T11:20:21.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:28.126 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:28.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.310+0000 7fb5fa01e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:28.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.310+0000 7fb5fa01e640 10 monclient: build_initial_monmap 2024-03-29T11:20:28.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.310+0000 7fb5fa01e640 10 monclient: monmap: 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:28.311715+0000 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:28.311715+0000 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:28.289 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:28.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.312+0000 7fb5fa01e640 1 Processor -- start 2024-03-29T11:20:28.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.312+0000 7fb5fa01e640 1 -- start start 2024-03-29T11:20:28.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.312+0000 7fb5fa01e640 10 monclient: init 2024-03-29T11:20:28.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:28.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:28.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ec006410 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:28.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 10 monclient(hunting): picked mon.noname-c con 0x7fb5ec02a640 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:28.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 10 monclient(hunting): picked mon.noname-a con 0x7fb5ec02b370 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:28.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec02a120 con 0x7fb5ec02b370 2024-03-29T11:20:28.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec0c75b0 con 0x7fb5ec02a640 2024-03-29T11:20:28.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec033110 con 0x7fb5ec006410 2024-03-29T11:20:28.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:28.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.313+0000 7fb5fa01e640 10 monclient(hunting): authenticate will time out at 849.622719s 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.317+0000 7fb5f37fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fb5ec006410 0x7fb5ec0345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:45819/0 (socket says 172.21.15.138:45819) 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.317+0000 7fb5f37fe640 1 -- 172.21.15.138:0/3886377487 learned_addr learned my addr 172.21.15.138:0/3886377487 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.318+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757344930 0 0) 0x7fb5ec033110 con 0x7fb5ec006410 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.318+0000 7fb5f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.318+0000 7fb5f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.318+0000 7fb5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.318+0000 7fb5f27fc640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.318+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e4003520 con 0x7fb5ec006410 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858688569 0 0) 0x7fb5ec02a120 con 0x7fb5ec02b370 2024-03-29T11:20:28.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5ec033110 con 0x7fb5ec02b370 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113089249 0 0) 0x7fb5e4003520 con 0x7fb5ec006410 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 >> v1:172.21.15.152:6789/0 conn(0x7fb5ec02a640 legacy=0x7fb5ec0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 >> v1:172.21.15.138:6789/0 conn(0x7fb5ec02b370 legacy=0x7fb5ec0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 1 monclient: found mon.noname-b 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec0332a0 con 0x7fb5ec006410 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:28.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.319+0000 7fb5f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5fa01e640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5fa01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb5e0001690 con 0x7fb5ec006410 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 10 monclient: dump: 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.300 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5e00018b0 con 0x7fb5ec006410 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 1 -- 172.21.15.138:0/3886377487 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb5e0001c80 con 0x7fb5ec006410 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5f27fc640 10 monclient: dump: 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.301 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5fa01e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:28.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.320+0000 7fb5fa01e640 4 set_mon_vals no callback set 2024-03-29T11:20:28.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.321+0000 7fb5fa01e640 10 monclient: shutdown 2024-03-29T11:20:28.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.321+0000 7fb5fa01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:28.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.321+0000 7fb5fa01e640 1 -- 172.21.15.138:0/3886377487 >> v1:172.21.15.138:6790/0 conn(0x7fb5ec006410 legacy=0x7fb5ec0345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.323+0000 7fb5fa01e640 1 -- 172.21.15.138:0/3886377487 shutdown_connections 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.323+0000 7fb5fa01e640 1 -- 172.21.15.138:0/3886377487 >> 172.21.15.138:0/3886377487 conn(0x7fb5ec0058f0 msgr2=0x7fb5ec005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.324+0000 7fb5fa01e640 1 -- 172.21.15.138:0/3886377487 shutdown_connections 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.324+0000 7fb5fa01e640 1 -- 172.21.15.138:0/3886377487 wait complete. 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.326+0000 7fb5fa01e640 10 monclient: build_initial_monmap 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.326+0000 7fb5fa01e640 10 monclient: monmap: 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:28.304 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:28.305 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:28.305 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:28.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.326+0000 7fb5fa01e640 1 librados: starting msgr at 2024-03-29T11:20:28.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.326+0000 7fb5fa01e640 1 librados: starting objecter 2024-03-29T11:20:28.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.326+0000 7fb5fa01e640 1 Processor -- start 2024-03-29T11:20:28.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.327+0000 7fb5fa01e640 1 -- start start 2024-03-29T11:20:28.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.327+0000 7fb5fa01e640 1 librados: setting wanted keys 2024-03-29T11:20:28.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.327+0000 7fb5fa01e640 1 librados: calling monclient init 2024-03-29T11:20:28.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.327+0000 7fb5fa01e640 10 monclient: init 2024-03-29T11:20:28.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.328+0000 7fb5fa01e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:28.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.328+0000 7fb5fa01e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.329+0000 7fb5fa01e640 10 monclient(hunting): picked mon.noname-c con 0x7fb5ec006410 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.329+0000 7fb5fa01e640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ec02a640 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.331+0000 7fb5f37fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fb5ec006410 0x7fb5ec02fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:45907/0 (socket says 172.21.15.138:45907) 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5f37fe640 1 -- 172.21.15.138:0/4084682960 learned_addr learned my addr 172.21.15.138:0/4084682960 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5fa01e640 10 monclient(hunting): picked mon.noname-a con 0x7fb5ec02b370 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5dffff640 10 client.?.objecter ms_handle_connect 0x7fb5ec006410 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5fa01e640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec160fa0 con 0x7fb5ec02b370 2024-03-29T11:20:28.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5fa01e640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec02ccd0 con 0x7fb5ec02a640 2024-03-29T11:20:28.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5fa01e640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec02f8d0 con 0x7fb5ec006410 2024-03-29T11:20:28.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5fa01e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:28.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.332+0000 7fb5fa01e640 10 monclient(hunting): authenticate will time out at 849.641655s 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998857600 0 0) 0x7fb5ec02f8d0 con 0x7fb5ec006410 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8004eb0 con 0x7fb5ec006410 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 client.?.objecter ms_handle_connect 0x7fb5ec02b370 2024-03-29T11:20:28.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2223992 0 0) 0x7fb5d8004eb0 con 0x7fb5ec006410 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5ec02f8d0 con 0x7fb5ec006410 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb5e0001610 con 0x7fb5ec006410 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): dump: 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:28.312 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198635149 0 0) 0x7fb5ec160fa0 con 0x7fb5ec02b370 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:20:28.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.333+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8004eb0 con 0x7fb5ec02b370 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3049156169 0 0) 0x7fb5ec02f8d0 con 0x7fb5ec006410 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 >> v1:172.21.15.152:6789/0 conn(0x7fb5ec02a640 legacy=0x7fb5ec161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 >> v1:172.21.15.138:6789/0 conn(0x7fb5ec02b370 legacy=0x7fb5ec16bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 1 monclient: found mon.c 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec0c1720 con 0x7fb5ec006410 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:20:28.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 10 monclient: _renew_subs 2024-03-29T11:20:28.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:28.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ec030510 con 0x7fb5ec006410 2024-03-29T11:20:28.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:28.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 10 monclient: _renew_subs 2024-03-29T11:20:28.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:28.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ec0307c0 con 0x7fb5ec006410 2024-03-29T11:20:28.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.334+0000 7fb5fa01e640 1 librados: init done 2024-03-29T11:20:28.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.336+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5e0004a90 con 0x7fb5ec006410 2024-03-29T11:20:28.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.336+0000 7fb5dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:28.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.336+0000 7fb5f1ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:28.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.337+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb5e0004ee0 con 0x7fb5ec006410 2024-03-29T11:20:28.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.337+0000 7fb5dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.337+0000 7fb5dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.337+0000 7fb5dffff640 10 monclient: dump: 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.337+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fb5e0005100 con 0x7fb5ec006410 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5ddffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5ddffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5cc005080 con 0x7fb5ec006410 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fb5e0084ed0 con 0x7fb5ec006410 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5dffff640 10 client.4250.objecter ms_dispatch 0x7fb5f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5dffff640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5dffff640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5dffff640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.338+0000 7fb5f2ffd640 10 monclient: get_auth_request con 0x7fb5d806dc20 auth_method 0 2024-03-29T11:20:28.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.339+0000 7fb5dffff640 10 client.4250.objecter ms_handle_connect 0x7fb5d806dc20 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.349+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5e0050830 con 0x7fb5ec006410 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.349+0000 7fb5dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.349+0000 7fb5dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.672+0000 7fb5ddffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.672+0000 7fb5ddffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.672+0000 7fb5ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.672+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5cc005330 con 0x7fb5ec006410 2024-03-29T11:20:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.676+0000 7fb5dffff640 1 -- 172.21.15.138:0/4084682960 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fb5e0052760 con 0x7fb5ec006410 2024-03-29T11:20:28.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.676+0000 7fb5dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.676+0000 7fb5dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:28.654 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:28.654 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.680+0000 7fb5ddffb640 10 librados: watch_flush enter 2024-03-29T11:20:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.680+0000 7fb5ddffb640 10 librados: watch_flush exit 2024-03-29T11:20:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.680+0000 7fb5ddffb640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T11:20:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.680+0000 7fb5ddffb640 10 client.4250.objecter successfully canceled tick 2024-03-29T11:20:28.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.682+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 >> v1:172.21.15.152:6816/3052585572 conn(0x7fb5d806dc20 legacy=0x7fb5d80700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.682+0000 7fb5ddffb640 10 monclient: shutdown 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.682+0000 7fb5ddffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.682+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 >> v1:172.21.15.138:6790/0 conn(0x7fb5ec006410 legacy=0x7fb5ec02fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.682+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 shutdown_connections 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.682+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 >> 172.21.15.138:0/4084682960 conn(0x7fb5ec0058f0 msgr2=0x7fb5ec0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.683+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 shutdown_connections 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.683+0000 7fb5ddffb640 1 -- 172.21.15.138:0/4084682960 wait complete. 2024-03-29T11:20:28.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:28.683+0000 7fb5ddffb640 1 librados: shutdown 2024-03-29T11:20:28.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:35.489 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:35.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.663+0000 7efdb7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.663+0000 7efdb7fff640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.663+0000 7efdb7fff640 10 monclient: monmap: 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:35.664855+0000 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:35.664855+0000 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:35.641 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:35.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.665+0000 7efdb7fff640 1 Processor -- start 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.666+0000 7efdb7fff640 1 -- start start 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.666+0000 7efdb7fff640 10 monclient: init 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 10 monclient(hunting): picked mon.noname-c con 0x7efdb0006d80 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 10 monclient(hunting): picked mon.noname-b con 0x7efdb00cc310 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 10 monclient(hunting): picked mon.noname-a con 0x7efdb00c9430 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb0008470 con 0x7efdb00c9430 2024-03-29T11:20:35.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb002be30 con 0x7efdb0006d80 2024-03-29T11:20:35.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb002c110 con 0x7efdb00cc310 2024-03-29T11:20:35.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.667+0000 7efdb7fff640 10 monclient(hunting): authenticate will time out at 856.976457s 2024-03-29T11:20:35.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.670+0000 7efdb77fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7efdb00c9430 0x7efdb00d74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33105/0 (socket says 172.21.15.138:33105) 2024-03-29T11:20:35.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.670+0000 7efdb77fe640 1 -- 172.21.15.138:0/1674133562 learned_addr learned my addr 172.21.15.138:0/1674133562 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:35.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.670+0000 7efdb67fc640 1 --1- 172.21.15.138:0/1674133562 >> v1:172.21.15.138:6790/0 conn(0x7efdb00cc310 0x7efdb00c8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:57473/0 (socket says 172.21.15.138:57473) 2024-03-29T11:20:35.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348678940 0 0) 0x7efdb0008470 con 0x7efdb00c9430 2024-03-29T11:20:35.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:20:35.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda0003520 con 0x7efdb00c9430 2024-03-29T11:20:35.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541118895 0 0) 0x7efdb002c110 con 0x7efdb00cc310 2024-03-29T11:20:35.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.671+0000 7efdb5ffb640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:20:35.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.672+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb0008470 con 0x7efdb00cc310 2024-03-29T11:20:35.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3636777655 0 0) 0x7efda0003520 con 0x7efdb00c9430 2024-03-29T11:20:35.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 >> v1:172.21.15.138:6790/0 conn(0x7efdb00cc310 legacy=0x7efdb00c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 >> v1:172.21.15.152:6789/0 conn(0x7efdb0006d80 legacy=0x7efdb00cbc00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:20:35.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb002c3f0 con 0x7efdb00c9430 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efdac0016e0 con 0x7efdb00c9430 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.675+0000 7efdb5ffb640 10 monclient: dump: 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:35.655 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.676+0000 7efdb7fff640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.676+0000 7efdb7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdac001840 con 0x7efdb00c9430 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb5ffb640 1 -- 172.21.15.138:0/1674133562 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efdac001c10 con 0x7efdb00c9430 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb5ffb640 10 monclient: dump: 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.677+0000 7efdb7fff640 4 set_mon_vals no callback set 2024-03-29T11:20:35.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.678+0000 7efdb7fff640 10 monclient: shutdown 2024-03-29T11:20:35.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.678+0000 7efdb7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.678+0000 7efdb7fff640 1 -- 172.21.15.138:0/1674133562 >> v1:172.21.15.138:6789/0 conn(0x7efdb00c9430 legacy=0x7efdb00d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.679+0000 7efdb7fff640 1 -- 172.21.15.138:0/1674133562 shutdown_connections 2024-03-29T11:20:35.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.679+0000 7efdb7fff640 1 -- 172.21.15.138:0/1674133562 >> 172.21.15.138:0/1674133562 conn(0x7efdb0005c00 msgr2=0x7efdb0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.682+0000 7efdb7fff640 1 -- 172.21.15.138:0/1674133562 shutdown_connections 2024-03-29T11:20:35.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.683+0000 7efdb7fff640 1 -- 172.21.15.138:0/1674133562 wait complete. 2024-03-29T11:20:35.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.684+0000 7efdb7fff640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.684+0000 7efdb7fff640 10 monclient: monmap: 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.685+0000 7efdb7fff640 1 librados: starting msgr at 2024-03-29T11:20:35.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.685+0000 7efdb7fff640 1 librados: starting objecter 2024-03-29T11:20:35.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.685+0000 7efdb7fff640 1 Processor -- start 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.685+0000 7efdb7fff640 1 -- start start 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.685+0000 7efdb7fff640 1 librados: setting wanted keys 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.685+0000 7efdb7fff640 1 librados: calling monclient init 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.685+0000 7efdb7fff640 10 monclient: init 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.686+0000 7efdb7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.686+0000 7efdb7fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.686+0000 7efdb7fff640 10 monclient(hunting): picked mon.noname-b con 0x7efdb0006d80 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:35.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.686+0000 7efdb7fff640 10 monclient(hunting): picked mon.noname-a con 0x7efdb00c9430 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:35.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.687+0000 7efdb7fff640 10 monclient(hunting): picked mon.noname-c con 0x7efdb00cc310 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.690+0000 7efdb6ffd640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7efdb0006d80 0x7efdb002c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:36985/0 (socket says 172.21.15.138:36985) 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.690+0000 7efdb6ffd640 1 -- 172.21.15.138:0/2150314017 learned_addr learned my addr 172.21.15.138:0/2150314017 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.692+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb0164610 con 0x7efdb00c9430 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.692+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb00c6270 con 0x7efdb0006d80 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.692+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb0163fd0 con 0x7efdb00cc310 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.692+0000 7efdb7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.692+0000 7efdb7fff640 10 monclient(hunting): authenticate will time out at 857.001362s 2024-03-29T11:20:35.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.692+0000 7efd9ffff640 10 client.?.objecter ms_handle_connect 0x7efdb0006d80 2024-03-29T11:20:35.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 client.?.objecter ms_handle_connect 0x7efdb00cc310 2024-03-29T11:20:35.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600595995 0 0) 0x7efdb0163fd0 con 0x7efdb00cc310 2024-03-29T11:20:35.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:20:35.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda4004eb0 con 0x7efdb00cc310 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911111981 0 0) 0x7efdb00c6270 con 0x7efdb0006d80 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.693+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb0163fd0 con 0x7efdb0006d80 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4135308485 0 0) 0x7efda4004eb0 con 0x7efdb00cc310 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdb00c6270 con 0x7efdb00cc310 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efdac001b50 con 0x7efdb00cc310 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:35.674 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 251363690 0 0) 0x7efdb0163fd0 con 0x7efdb0006d80 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda4004eb0 con 0x7efdb0006d80 2024-03-29T11:20:35.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1647556777 0 0) 0x7efdb00c6270 con 0x7efdb00cc310 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 >> v1:172.21.15.152:6789/0 conn(0x7efdb0006d80 legacy=0x7efdb002c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.694+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 >> v1:172.21.15.138:6789/0 conn(0x7efdb00c9430 legacy=0x7efdb00c5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 1 monclient: found mon.c 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb0164160 con 0x7efdb00cc310 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efd9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 10 monclient: _renew_subs 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:35.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb00cb3e0 con 0x7efdb00cc310 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 10 monclient: _renew_subs 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb0155e50 con 0x7efdb00cc310 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.695+0000 7efdb7fff640 1 librados: init done 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.696+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdac001cb0 con 0x7efdb00cc310 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.696+0000 7efd9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.696+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7efdac004ae0 con 0x7efdb00cc310 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.696+0000 7efd9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.696+0000 7efd9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.696+0000 7efd9ffff640 10 monclient: dump: 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:35.677 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:35.678 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.696+0000 7efdb4ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.698+0000 7efdb7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.698+0000 7efdb7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.698+0000 7efdb7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.698+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd98005080 con 0x7efdb00cc310 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.700+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7efdac01aab0 con 0x7efdb00cc310 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.701+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7efdac084a60 con 0x7efdb00cc310 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.701+0000 7efdb6ffd640 10 monclient: get_auth_request con 0x7efda406ded0 auth_method 0 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.701+0000 7efd9ffff640 10 client.4256.objecter ms_dispatch 0x7efdb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:35.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.701+0000 7efd9ffff640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:35.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.701+0000 7efd9ffff640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:35.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.701+0000 7efd9ffff640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.701+0000 7efd9ffff640 10 client.4256.objecter ms_handle_connect 0x7efda406ded0 2024-03-29T11:20:35.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.711+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdac050170 con 0x7efdb00cc310 2024-03-29T11:20:35.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.711+0000 7efd9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:35.711+0000 7efd9ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:36.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.030+0000 7efdb7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.030+0000 7efdb7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.030+0000 7efdb7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:36.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.030+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd98005330 con 0x7efdb00cc310 2024-03-29T11:20:36.008 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:36.008 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:36.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.031+0000 7efd9ffff640 1 -- 172.21.15.138:0/2150314017 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7efdac052270 con 0x7efdb00cc310 2024-03-29T11:20:36.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.031+0000 7efd9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.031+0000 7efd9ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:36.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.035+0000 7efdb7fff640 10 librados: watch_flush enter 2024-03-29T11:20:36.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.036+0000 7efdb7fff640 10 librados: watch_flush exit 2024-03-29T11:20:36.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.036+0000 7efdb7fff640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:20:36.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.036+0000 7efdb7fff640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:20:36.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 >> v1:172.21.15.152:6816/3052585572 conn(0x7efda406ded0 legacy=0x7efda4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 10 monclient: shutdown 2024-03-29T11:20:36.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:36.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 >> v1:172.21.15.138:6790/0 conn(0x7efdb00cc310 legacy=0x7efdb00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 shutdown_connections 2024-03-29T11:20:36.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 >> 172.21.15.138:0/2150314017 conn(0x7efdb0005c00 msgr2=0x7efdb0039ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:36.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 shutdown_connections 2024-03-29T11:20:36.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.038+0000 7efdb7fff640 1 -- 172.21.15.138:0/2150314017 wait complete. 2024-03-29T11:20:36.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:36.039+0000 7efdb7fff640 1 librados: shutdown 2024-03-29T11:20:36.037 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:42.845 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:42.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.016+0000 7f36a24db640 10 monclient: get_monmap_and_config 2024-03-29T11:20:42.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.016+0000 7f36a24db640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.016+0000 7f36a24db640 10 monclient: monmap: 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:43.017333+0000 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:43.017333+0000 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:42.994 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.017+0000 7f36a24db640 1 Processor -- start 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.017+0000 7f36a24db640 1 -- start start 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.017+0000 7f36a24db640 10 monclient: init 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 10 monclient(hunting): picked mon.noname-b con 0x7f369c0a9750 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 10 monclient(hunting): picked mon.noname-c con 0x7f369c0ae220 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 10 monclient(hunting): picked mon.noname-a con 0x7f369c13b9d0 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c142490 con 0x7f369c13b9d0 2024-03-29T11:20:42.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c0ac130 con 0x7f369c0ae220 2024-03-29T11:20:42.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c0ac410 con 0x7f369c0a9750 2024-03-29T11:20:42.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.018+0000 7f36a24db640 10 monclient(hunting): authenticate will time out at 864.327227s 2024-03-29T11:20:42.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.021+0000 7f36a0cd8640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f369c0ae220 0x7f369c13b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:57835/0 (socket says 172.21.15.138:57835) 2024-03-29T11:20:42.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.021+0000 7f36a0cd8640 1 -- 172.21.15.138:0/1542315424 learned_addr learned my addr 172.21.15.138:0/1542315424 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:43.002 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.022+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349437214 0 0) 0x7f369c0ac130 con 0x7f369c0ae220 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.022+0000 7f3693fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.022+0000 7f3693fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.022+0000 7f3693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.022+0000 7f3693fff640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.022+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3688003520 con 0x7f369c0ae220 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424573943 0 0) 0x7f369c0ac410 con 0x7f369c0a9750 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c0ac130 con 0x7f369c0a9750 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1791521482 0 0) 0x7f3688003520 con 0x7f369c0ae220 2024-03-29T11:20:43.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 >> v1:172.21.15.138:6790/0 conn(0x7f369c0a9750 legacy=0x7f369c141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 >> v1:172.21.15.138:6789/0 conn(0x7f369c13b9d0 legacy=0x7f369c149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 monclient: found mon.noname-c 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c0afa40 con 0x7f369c0ae220 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f368c001690 con 0x7f369c0ae220 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:43.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.023+0000 7f3693fff640 10 monclient: dump: 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.024+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f368c0017f0 con 0x7f369c0ae220 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.024+0000 7f3693fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.024+0000 7f3693fff640 1 -- 172.21.15.138:0/1542315424 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f368c001b80 con 0x7f369c0ae220 2024-03-29T11:20:43.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.024+0000 7f3693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.024+0000 7f3693fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.024+0000 7f3693fff640 10 monclient: dump: 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.025+0000 7f36a24db640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:20:43.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.025+0000 7f36a24db640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:43.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.025+0000 7f36a24db640 4 set_mon_vals no callback set 2024-03-29T11:20:43.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.029+0000 7f36a24db640 10 monclient: shutdown 2024-03-29T11:20:43.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.029+0000 7f36a24db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:43.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.029+0000 7f36a24db640 1 -- 172.21.15.138:0/1542315424 >> v1:172.21.15.152:6789/0 conn(0x7f369c0ae220 legacy=0x7f369c13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.029+0000 7f36a24db640 1 -- 172.21.15.138:0/1542315424 shutdown_connections 2024-03-29T11:20:43.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.029+0000 7f36a24db640 1 -- 172.21.15.138:0/1542315424 >> 172.21.15.138:0/1542315424 conn(0x7f369c0673e0 msgr2=0x7f369c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:43.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.032+0000 7f36a24db640 1 -- 172.21.15.138:0/1542315424 shutdown_connections 2024-03-29T11:20:43.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.034+0000 7f36a24db640 1 -- 172.21.15.138:0/1542315424 wait complete. 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.035+0000 7f36a24db640 10 monclient: build_initial_monmap 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.035+0000 7f36a24db640 10 monclient: monmap: 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:43.012 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:43.013 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:43.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.035+0000 7f36a24db640 1 librados: starting msgr at 2024-03-29T11:20:43.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.035+0000 7f36a24db640 1 librados: starting objecter 2024-03-29T11:20:43.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.035+0000 7f36a24db640 1 Processor -- start 2024-03-29T11:20:43.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.039+0000 7f36a24db640 1 -- start start 2024-03-29T11:20:43.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.039+0000 7f36a24db640 1 librados: setting wanted keys 2024-03-29T11:20:43.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.039+0000 7f36a24db640 1 librados: calling monclient init 2024-03-29T11:20:43.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.039+0000 7f36a24db640 10 monclient: init 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 10 monclient(hunting): picked mon.noname-c con 0x7f369c0a9750 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 10 monclient(hunting): picked mon.noname-a con 0x7f369c0ae220 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 10 monclient(hunting): picked mon.noname-b con 0x7f369c13b9d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c072770 con 0x7f369c0ae220 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c1df9a0 con 0x7f369c13b9d0 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c1df570 con 0x7f369c0a9750 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:43.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.040+0000 7f36a24db640 10 monclient(hunting): authenticate will time out at 864.349710s 2024-03-29T11:20:43.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.044+0000 7f36a0cd8640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f369c0ae220 0x7f369c070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:39095/0 (socket says 172.21.15.138:39095) 2024-03-29T11:20:43.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.044+0000 7f36a0cd8640 1 -- 172.21.15.138:0/787436386 learned_addr learned my addr 172.21.15.138:0/787436386 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:43.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.047+0000 7f3691ffb640 10 client.?.objecter ms_handle_connect 0x7f369c0a9750 2024-03-29T11:20:43.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 client.?.objecter ms_handle_connect 0x7f369c0ae220 2024-03-29T11:20:43.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 client.?.objecter ms_handle_connect 0x7f369c13b9d0 2024-03-29T11:20:43.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175735233 0 0) 0x7f369c1df570 con 0x7f369c0a9750 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3680004eb0 con 0x7f369c0a9750 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596362485 0 0) 0x7f369c1df9a0 con 0x7f369c13b9d0 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c1df570 con 0x7f369c13b9d0 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1046107746 0 0) 0x7f3680004eb0 con 0x7f369c0a9750 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f369c1df9a0 con 0x7f369c0a9750 2024-03-29T11:20:43.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f3698001970 con 0x7f369c0a9750 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:43.033 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926446473 0 0) 0x7f369c1df570 con 0x7f369c13b9d0 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3680004eb0 con 0x7f369c13b9d0 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.048+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f3694001a40 con 0x7f369c13b9d0 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:43.034 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4096605031 0 0) 0x7f369c1df9a0 con 0x7f369c0a9750 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 >> v1:172.21.15.152:6789/0 conn(0x7f369c13b9d0 legacy=0x7f369c071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 >> v1:172.21.15.138:6789/0 conn(0x7f369c0ae220 legacy=0x7f369c070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 1 monclient: found mon.c 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c1df700 con 0x7f369c0a9750 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:43.035 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f3691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 10 monclient: _renew_subs 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c140ba0 con 0x7f369c0a9750 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 10 monclient: _renew_subs 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c140ea0 con 0x7f369c0a9750 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.049+0000 7f36a24db640 1 librados: init done 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.050+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3698003610 con 0x7f369c0a9750 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.050+0000 7f3691ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:43.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.050+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f3698003920 con 0x7f369c0a9750 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.050+0000 7f3691ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.050+0000 7f3691ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.050+0000 7f3691ffb640 10 monclient: dump: 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.050+0000 7f3692ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:43.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.052+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f36980198f0 con 0x7f369c0a9750 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.052+0000 7f36a24db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.052+0000 7f36a24db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.052+0000 7f36a24db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.052+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c005080 con 0x7f369c0a9750 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.053+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f3698083890 con 0x7f369c0a9750 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.053+0000 7f3691ffb640 10 client.4262.objecter ms_dispatch 0x7f369c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.053+0000 7f3691ffb640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.053+0000 7f3691ffb640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.053+0000 7f3691ffb640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:20:43.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.053+0000 7f36a0cd8640 10 monclient: get_auth_request con 0x7f368006e040 auth_method 0 2024-03-29T11:20:43.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.056+0000 7f3691ffb640 10 client.4262.objecter ms_handle_connect 0x7f368006e040 2024-03-29T11:20:43.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.061+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f369804f170 con 0x7f369c0a9750 2024-03-29T11:20:43.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.061+0000 7f3691ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.061+0000 7f3691ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:43.369 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.392+0000 7f36a24db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.369 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.392+0000 7f36a24db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.369 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.392+0000 7f36a24db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:20:43.369 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.392+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f367c005330 con 0x7f369c0a9750 2024-03-29T11:20:43.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.392+0000 7f3691ffb640 1 -- 172.21.15.138:0/787436386 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f36980510a0 con 0x7f369c0a9750 2024-03-29T11:20:43.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.392+0000 7f3691ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.392+0000 7f3691ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:43.370 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:43.370 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:43.372 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.396+0000 7f36a24db640 10 librados: watch_flush enter 2024-03-29T11:20:43.373 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.396+0000 7f36a24db640 10 librados: watch_flush exit 2024-03-29T11:20:43.373 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.396+0000 7f36a24db640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:20:43.373 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.396+0000 7f36a24db640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:20:43.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.397+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 >> v1:172.21.15.152:6816/3052585572 conn(0x7f368006e040 legacy=0x7f3680070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.397+0000 7f36a24db640 10 monclient: shutdown 2024-03-29T11:20:43.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.397+0000 7f36a24db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:43.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.397+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 >> v1:172.21.15.138:6790/0 conn(0x7f369c0a9750 legacy=0x7f369c1dee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.398+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 shutdown_connections 2024-03-29T11:20:43.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.398+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 >> 172.21.15.138:0/787436386 conn(0x7f369c0673e0 msgr2=0x7f369c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:43.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.398+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 shutdown_connections 2024-03-29T11:20:43.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.398+0000 7f36a24db640 1 -- 172.21.15.138:0/787436386 wait complete. 2024-03-29T11:20:43.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:43.398+0000 7f36a24db640 1 librados: shutdown 2024-03-29T11:20:43.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:50.203 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:50.362 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.386+0000 7f0c1b90d640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.362 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.386+0000 7f0c1b90d640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.362 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.386+0000 7f0c1b90d640 10 monclient: monmap: 2024-03-29T11:20:50.362 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:50.362 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:50.362 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:50.386445+0000 2024-03-29T11:20:50.362 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:50.386445+0000 2024-03-29T11:20:50.363 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.363 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:50.363 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:50.363 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:50.363 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:50.363 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.387+0000 7f0c1b90d640 1 Processor -- start 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.387+0000 7f0c1b90d640 1 -- start start 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.387+0000 7f0c1b90d640 10 monclient: init 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 10 monclient(hunting): picked mon.noname-a con 0x7f0c140af290 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c14143ec0 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 10 monclient(hunting): picked mon.noname-c con 0x7f0c140af670 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c140ae740 con 0x7f0c140af290 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c140b06a0 con 0x7f0c140af670 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c140b0980 con 0x7f0c14143ec0 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.367 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.388+0000 7f0c1b90d640 10 monclient(hunting): authenticate will time out at 871.696216s 2024-03-29T11:20:50.368 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.390+0000 7f0c19eaa640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f0c140af670 0x7f0c14151dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:46555/0 (socket says 172.21.15.138:46555) 2024-03-29T11:20:50.368 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.390+0000 7f0c19eaa640 1 -- 172.21.15.138:0/1370685141 learned_addr learned my addr 172.21.15.138:0/1370685141 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:50.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.395+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205165811 0 0) 0x7f0c140b06a0 con 0x7f0c140af670 2024-03-29T11:20:50.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.395+0000 7f0c0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.395+0000 7f0c0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.395+0000 7f0c0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.395+0000 7f0c0bfff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:20:50.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.395+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c00003520 con 0x7f0c140af670 2024-03-29T11:20:50.373 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.396+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1192552616 0 0) 0x7f0c00003520 con 0x7f0c140af670 2024-03-29T11:20:50.373 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.396+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 >> v1:172.21.15.138:6790/0 conn(0x7f0c14143ec0 legacy=0x7f0c140b40d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:50.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 >> v1:172.21.15.138:6789/0 conn(0x7f0c140af290 legacy=0x7f0c141437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 1 monclient: found mon.noname-c 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140b0c60 con 0x7f0c140af670 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0c0c001a10 con 0x7f0c140af670 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: dump: 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:50.376 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c1b90d640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:20:50.377 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c1b90d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 1 -- 172.21.15.138:0/1370685141 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c0c001be0 con 0x7f0c140af670 2024-03-29T11:20:50.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c1b90d640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.398+0000 7f0c1b90d640 4 set_mon_vals no callback set 2024-03-29T11:20:50.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.399+0000 7f0c1b90d640 10 monclient: shutdown 2024-03-29T11:20:50.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.399+0000 7f0c1b90d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.399+0000 7f0c1b90d640 1 -- 172.21.15.138:0/1370685141 >> v1:172.21.15.152:6789/0 conn(0x7f0c140af670 legacy=0x7f0c14151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.403+0000 7f0c1b90d640 1 -- 172.21.15.138:0/1370685141 shutdown_connections 2024-03-29T11:20:50.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.403+0000 7f0c1b90d640 1 -- 172.21.15.138:0/1370685141 >> 172.21.15.138:0/1370685141 conn(0x7f0c14066a00 msgr2=0x7f0c14066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.403+0000 7f0c1b90d640 1 -- 172.21.15.138:0/1370685141 shutdown_connections 2024-03-29T11:20:50.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.403+0000 7f0c1b90d640 1 -- 172.21.15.138:0/1370685141 wait complete. 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.404+0000 7f0c1b90d640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.404+0000 7f0c1b90d640 10 monclient: monmap: 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.383 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 1 librados: starting msgr at 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 1 librados: starting objecter 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 1 Processor -- start 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 1 -- start start 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 1 librados: setting wanted keys 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 1 librados: calling monclient init 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 10 monclient: init 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.405+0000 7f0c1b90d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 10 monclient(hunting): picked mon.noname-a con 0x7f0c140af290 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:50.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c140af670 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:50.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 10 monclient(hunting): picked mon.noname-c con 0x7f0c14143ec0 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:50.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c14146ca0 con 0x7f0c140af290 2024-03-29T11:20:50.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c141e4d80 con 0x7f0c140af670 2024-03-29T11:20:50.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c14071800 con 0x7f0c14143ec0 2024-03-29T11:20:50.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.406+0000 7f0c1b90d640 10 monclient(hunting): authenticate will time out at 871.714124s 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.411+0000 7f0c18ea8640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f0c140af670 0x7f0c140710f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:46685/0 (socket says 172.21.15.138:46685) 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.411+0000 7f0c18ea8640 1 -- 172.21.15.138:0/652861433 learned_addr learned my addr 172.21.15.138:0/652861433 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.411+0000 7f0c09ffb640 10 client.?.objecter ms_handle_connect 0x7f0c140af670 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.411+0000 7f0c09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287579111 0 0) 0x7f0c141e4d80 con 0x7f0c140af670 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf8004eb0 con 0x7f0c140af670 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 751031510 0 0) 0x7f0bf8004eb0 con 0x7f0c140af670 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c141e4d80 con 0x7f0c140af670 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0c04001890 con 0x7f0c140af670 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.412+0000 7f0c09ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:50.396 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c19eaa640 1 -- 172.21.15.138:0/652861433 >> v1:172.21.15.138:6790/0 conn(0x7f0c14143ec0 legacy=0x7f0c141e3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c19eaa640 1 -- 172.21.15.138:0/652861433 >> v1:172.21.15.138:6790/0 conn(0x7f0c14143ec0 legacy=0x7f0c141e3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c19eaa640 1 --1- 172.21.15.138:0/652861433 >> v1:172.21.15.138:6790/0 conn(0x7f0c14143ec0 0x7f0c141e3e50 :-1 s=OPENED pgs=85160 cs=1 l=1).handle_message read tag failed 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c19eaa640 1 --1- 172.21.15.138:0/652861433 >> v1:172.21.15.138:6790/0 conn(0x7f0c14143ec0 0x7f0c141e3e50 :-1 s=OPENED pgs=85160 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 10 client.?.objecter ms_handle_connect 0x7f0c14143ec0 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.138:6790/0 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2252849807 0 0) 0x7f0c141e4d80 con 0x7f0c140af670 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 >> v1:172.21.15.138:6790/0 conn(0x7f0c14143ec0 legacy=0x7f0c141e3e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:20:50.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.413+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 >> v1:172.21.15.138:6789/0 conn(0x7f0c140af290 legacy=0x7f0c14146590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 1 monclient: found mon.b 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c141e4560 con 0x7f0c140af670 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 10 monclient: _renew_subs 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c140709f0 con 0x7f0c140af670 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 10 monclient: _renew_subs 2024-03-29T11:20:50.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1406ed70 con 0x7f0c140af670 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.414+0000 7f0c1b90d640 1 librados: init done 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.416+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c04001af0 con 0x7f0c140af670 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.416+0000 7f0c09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.416+0000 7f0c0affd640 4 set_mon_vals no callback set 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.417+0000 7f0c1b90d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.417+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0c04003a60 con 0x7f0c140af670 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.417+0000 7f0c1b90d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.418+0000 7f0c1b90d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.418+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf4005080 con 0x7f0c140af670 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.419+0000 7f0c09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.419+0000 7f0c09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.419+0000 7f0c09ffb640 10 monclient: dump: 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:50.399 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:50.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.419+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f0c04003c80 con 0x7f0c140af670 2024-03-29T11:20:50.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.420+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f0c04084b80 con 0x7f0c140af670 2024-03-29T11:20:50.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.420+0000 7f0c09ffb640 10 client.4249.objecter ms_dispatch 0x7f0c14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:50.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.420+0000 7f0c09ffb640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:50.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.420+0000 7f0c09ffb640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:50.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.420+0000 7f0c09ffb640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.420+0000 7f0c196a9640 10 monclient: get_auth_request con 0x7f0bf806da70 auth_method 0 2024-03-29T11:20:50.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.421+0000 7f0c09ffb640 10 client.4249.objecter ms_handle_connect 0x7f0bf806da70 2024-03-29T11:20:50.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.427+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c04050460 con 0x7f0c140af670 2024-03-29T11:20:50.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.427+0000 7f0c09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.427+0000 7f0c09ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.754+0000 7f0c1b90d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.754+0000 7f0c1b90d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.754+0000 7f0c1b90d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:50.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.754+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bf4005330 con 0x7f0c140af670 2024-03-29T11:20:50.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.758+0000 7f0c09ffb640 1 -- 172.21.15.138:0/652861433 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f0c04052390 con 0x7f0c140af670 2024-03-29T11:20:50.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.758+0000 7f0c09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.758+0000 7f0c09ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:50.734 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:50.734 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.761+0000 7f0c1b90d640 10 librados: watch_flush enter 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 10 librados: watch_flush exit 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 10 client.4249.objecter successfully canceled tick 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 >> v1:172.21.15.152:6816/3052585572 conn(0x7f0bf806da70 legacy=0x7f0bf806ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 10 monclient: shutdown 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 >> v1:172.21.15.152:6789/0 conn(0x7f0c140af670 legacy=0x7f0c140710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 shutdown_connections 2024-03-29T11:20:50.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 >> 172.21.15.138:0/652861433 conn(0x7f0c14066a00 msgr2=0x7f0c14141e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 shutdown_connections 2024-03-29T11:20:50.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 1 -- 172.21.15.138:0/652861433 wait complete. 2024-03-29T11:20:50.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:50.762+0000 7f0c1b90d640 1 librados: shutdown 2024-03-29T11:20:50.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:57.565 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.743+0000 7f74e3aba640 10 monclient: get_monmap_and_config 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.743+0000 7f74e3aba640 10 monclient: build_initial_monmap 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.743+0000 7f74e3aba640 10 monclient: monmap: 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:20:57.744584+0000 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:20:57.744584+0000 2024-03-29T11:20:57.720 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:57.721 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:57.721 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:57.721 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:20:57.721 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:20:57.721 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:57.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.745+0000 7f74e3aba640 1 Processor -- start 2024-03-29T11:20:57.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.746+0000 7f74e3aba640 1 -- start start 2024-03-29T11:20:57.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.746+0000 7f74e3aba640 10 monclient: init 2024-03-29T11:20:57.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.747+0000 7f74e3aba640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:57.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.747+0000 7f74e3aba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:57.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.747+0000 7f74e3aba640 10 monclient(hunting): picked mon.noname-c con 0x7f74dc0b0860 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:57.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.747+0000 7f74e3aba640 10 monclient(hunting): picked mon.noname-a con 0x7f74dc13ef30 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:57.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.747+0000 7f74e3aba640 10 monclient(hunting): picked mon.noname-b con 0x7f74dc0af8e0 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:57.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.749+0000 7f74e3aba640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0b0c40 con 0x7f74dc13ef30 2024-03-29T11:20:57.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.749+0000 7f74e3aba640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0aed10 con 0x7f74dc0b0860 2024-03-29T11:20:57.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.749+0000 7f74e3aba640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0afd50 con 0x7f74dc0af8e0 2024-03-29T11:20:57.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.750+0000 7f74e3aba640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:57.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.750+0000 7f74e3aba640 10 monclient(hunting): authenticate will time out at 879.059120s 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.752+0000 7f74e1055640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f74dc13ef30 0x7f74dc0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:49051/0 (socket says 172.21.15.138:49051) 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.752+0000 7f74e1055640 1 -- 172.21.15.138:0/228817604 learned_addr learned my addr 172.21.15.138:0/228817604 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853491791 0 0) 0x7f74dc0aed10 con 0x7f74dc0b0860 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c8003500 con 0x7f74dc0b0860 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206060212 0 0) 0x7f74dc0b0c40 con 0x7f74dc13ef30 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc0aed10 con 0x7f74dc13ef30 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786431169 0 0) 0x7f74c8003500 con 0x7f74dc0b0860 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 >> v1:172.21.15.138:6790/0 conn(0x7f74dc0af8e0 legacy=0x7f74dc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 >> v1:172.21.15.138:6789/0 conn(0x7f74dc13ef30 legacy=0x7f74dc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient: _finish_hunting 0 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 monclient: found mon.noname-c 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0ab1b0 con 0x7f74dc0b0860 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient: _finish_auth 0 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient: _check_auth_tickets 2024-03-29T11:20:57.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f74d80015c0 con 0x7f74dc0b0860 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.753+0000 7f74e0854640 10 monclient: dump: 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.734 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:57.735 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:57.735 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:57.735 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:57.735 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:57.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.755+0000 7f74e3aba640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:20:57.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.755+0000 7f74e3aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:57.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74d8001720 con 0x7f74dc0b0860 2024-03-29T11:20:57.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e0854640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:57.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e0854640 1 -- 172.21.15.138:0/228817604 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f74d8001af0 con 0x7f74dc0b0860 2024-03-29T11:20:57.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e0854640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e0854640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:57.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e0854640 10 monclient: dump: 2024-03-29T11:20:57.739 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:57.739 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:57.739 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.739 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.739 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.739 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:57.739 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e3aba640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.757+0000 7f74e3aba640 4 set_mon_vals no callback set 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.758+0000 7f74e3aba640 10 monclient: shutdown 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.758+0000 7f74e3aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:57.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.758+0000 7f74e3aba640 1 -- 172.21.15.138:0/228817604 >> v1:172.21.15.152:6789/0 conn(0x7f74dc0b0860 legacy=0x7f74dc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.760+0000 7f74e3aba640 1 -- 172.21.15.138:0/228817604 shutdown_connections 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.760+0000 7f74e3aba640 1 -- 172.21.15.138:0/228817604 >> 172.21.15.138:0/228817604 conn(0x7f74dc06b660 msgr2=0x7f74dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.762+0000 7f74e3aba640 1 -- 172.21.15.138:0/228817604 shutdown_connections 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.762+0000 7f74e3aba640 1 -- 172.21.15.138:0/228817604 wait complete. 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.763+0000 7f74e3aba640 10 monclient: build_initial_monmap 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.763+0000 7f74e3aba640 10 monclient: monmap: 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:20:57.744 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.763+0000 7f74e3aba640 1 librados: starting msgr at 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.763+0000 7f74e3aba640 1 librados: starting objecter 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.764+0000 7f74e3aba640 1 Processor -- start 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.764+0000 7f74e3aba640 1 -- start start 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.764+0000 7f74e3aba640 1 librados: setting wanted keys 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.764+0000 7f74e3aba640 1 librados: calling monclient init 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.764+0000 7f74e3aba640 10 monclient: init 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.764+0000 7f74e3aba640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:57.745 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.765+0000 7f74e3aba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.765+0000 7f74e3aba640 10 monclient(hunting): picked mon.noname-b con 0x7f74dc0af8e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.765+0000 7f74e3aba640 10 monclient(hunting): picked mon.noname-a con 0x7f74dc0b0860 addr v1:172.21.15.138:6789/0 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.766+0000 7f74e3aba640 10 monclient(hunting): picked mon.noname-c con 0x7f74dc13ef30 addr v1:172.21.15.138:6790/0 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.766+0000 7f74e3aba640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0b3c80 con 0x7f74dc0b0860 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.766+0000 7f74e3aba640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0b30e0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.766+0000 7f74e3aba640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0b2dc0 con 0x7f74dc13ef30 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.766+0000 7f74e3aba640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:57.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.766+0000 7f74e3aba640 10 monclient(hunting): authenticate will time out at 879.075132s 2024-03-29T11:20:57.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74e1856640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f74dc0af8e0 0x7f74dc1c4f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:58345/0 (socket says 172.21.15.138:58345) 2024-03-29T11:20:57.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74e1856640 1 -- 172.21.15.138:0/1800154587 learned_addr learned my addr 172.21.15.138:0/1800154587 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 client.?.objecter ms_handle_connect 0x7f74dc0af8e0 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 client.?.objecter ms_handle_connect 0x7f74dc13ef30 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912189462 0 0) 0x7f74dc0b30e0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:20:57.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.772+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c0004eb0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2111376881 0 0) 0x7f74c0004eb0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74dc0b30e0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f74d8001ab0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient(hunting): dump: 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:57.754 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 381945947 0 0) 0x7f74dc0b30e0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 >> v1:172.21.15.138:6790/0 conn(0x7f74dc13ef30 legacy=0x7f74dc132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 >> v1:172.21.15.138:6789/0 conn(0x7f74dc0b0860 legacy=0x7f74dc0b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:57.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 monclient: found mon.b 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc1c4550 con 0x7f74dc0af8e0 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.776+0000 7f74e3aba640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 10 monclient: _renew_subs 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc1c4830 con 0x7f74dc0af8e0 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 10 monclient: _renew_subs 2024-03-29T11:20:57.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:57.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc0b2620 con 0x7f74dc0af8e0 2024-03-29T11:20:57.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.777+0000 7f74e3aba640 1 librados: init done 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.778+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74d80038a0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.778+0000 7f74d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.778+0000 7f74d3fff640 4 set_mon_vals no callback set 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.778+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f74d80011a0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.778+0000 7f74d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.779+0000 7f74d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.779+0000 7f74d27fc640 10 monclient: dump: 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.758 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:20:57.759 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.759 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:20:57.759 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:20:57.759 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:20:57.759 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:20:57.759 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:20:57.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.782+0000 7f74e3aba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.782+0000 7f74e3aba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.783+0000 7f74e3aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:57.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.783+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c4005080 con 0x7f74dc0af8e0 2024-03-29T11:20:57.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.786+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f74d8003cb0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.786+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f74d8084bb0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.786+0000 7f74d27fc640 10 client.4255.objecter ms_dispatch 0x7f74dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:57.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.787+0000 7f74d27fc640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:57.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.787+0000 7f74d27fc640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:57.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.787+0000 7f74d27fc640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T11:20:57.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.787+0000 7f74e1055640 10 monclient: get_auth_request con 0x7f74c006db00 auth_method 0 2024-03-29T11:20:57.766 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.789+0000 7f74d27fc640 10 client.4255.objecter ms_handle_connect 0x7f74c006db00 2024-03-29T11:20:57.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.795+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74d80502c0 con 0x7f74dc0af8e0 2024-03-29T11:20:57.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.795+0000 7f74d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:57.795+0000 7f74d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:58.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.120+0000 7f74e3aba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.120+0000 7f74e3aba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.120+0000 7f74e3aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:20:58.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.120+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74c4005330 con 0x7f74dc0af8e0 2024-03-29T11:20:58.101 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.124+0000 7f74d27fc640 1 -- 172.21.15.138:0/1800154587 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f74d80523c0 con 0x7f74dc0af8e0 2024-03-29T11:20:58.101 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.124+0000 7f74d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.101 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.124+0000 7f74d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:58.102 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:20:58.102 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:58.105 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.128+0000 7f74e3aba640 10 librados: watch_flush enter 2024-03-29T11:20:58.105 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.128+0000 7f74e3aba640 10 librados: watch_flush exit 2024-03-29T11:20:58.105 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.128+0000 7f74e3aba640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T11:20:58.105 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.128+0000 7f74e3aba640 10 client.4255.objecter successfully canceled tick 2024-03-29T11:20:58.106 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.129+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 >> v1:172.21.15.152:6816/3052585572 conn(0x7f74c006db00 legacy=0x7f74c006ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.129+0000 7f74e3aba640 10 monclient: shutdown 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.129+0000 7f74e3aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.129+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 >> v1:172.21.15.152:6789/0 conn(0x7f74dc0af8e0 legacy=0x7f74dc1c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.130+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 shutdown_connections 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.130+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 >> 172.21.15.138:0/1800154587 conn(0x7f74dc06b660 msgr2=0x7f74dc13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.130+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 shutdown_connections 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.130+0000 7f74e3aba640 1 -- 172.21.15.138:0/1800154587 wait complete. 2024-03-29T11:20:58.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:20:58.130+0000 7f74e3aba640 1 librados: shutdown 2024-03-29T11:20:58.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:04.936 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:05.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.120+0000 7f96d4300640 10 monclient: get_monmap_and_config 2024-03-29T11:21:05.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.120+0000 7f96d4300640 10 monclient: build_initial_monmap 2024-03-29T11:21:05.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.120+0000 7f96d4300640 10 monclient: monmap: 2024-03-29T11:21:05.098 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:05.098 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:05.098 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:05.121793+0000 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:05.121793+0000 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.121+0000 7f96d4300640 1 Processor -- start 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.121+0000 7f96d4300640 1 -- start start 2024-03-29T11:21:05.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.121+0000 7f96d4300640 10 monclient: init 2024-03-29T11:21:05.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:05.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 10 monclient(hunting): picked mon.noname-b con 0x7f96cc0a96c0 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 10 monclient(hunting): picked mon.noname-c con 0x7f96cc0a9aa0 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 10 monclient(hunting): picked mon.noname-a con 0x7f96cc0ad730 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc13fee0 con 0x7f96cc0ad730 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc0aab80 con 0x7f96cc0a9aa0 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc145e80 con 0x7f96cc0a96c0 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:05.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.122+0000 7f96d4300640 10 monclient(hunting): authenticate will time out at 886.431752s 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.127+0000 7f96d289d640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f96cc0ad730 0x7f96cc14dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:56587/0 (socket says 172.21.15.138:56587) 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.127+0000 7f96d289d640 1 -- 172.21.15.138:0/4281783335 learned_addr learned my addr 172.21.15.138:0/4281783335 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.127+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216752224 0 0) 0x7f96cc13fee0 con 0x7f96cc0ad730 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.128+0000 7f96d109a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.128+0000 7f96d109a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.128+0000 7f96d109a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.128+0000 7f96d109a640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.128+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b4003520 con 0x7f96cc0ad730 2024-03-29T11:21:05.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3838471011 0 0) 0x7f96b4003520 con 0x7f96cc0ad730 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 >> v1:172.21.15.138:6790/0 conn(0x7f96cc0a96c0 legacy=0x7f96cc13f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 >> v1:172.21.15.152:6789/0 conn(0x7f96cc0a9aa0 legacy=0x7f96cc145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 1 monclient: found mon.noname-a 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0b4440 con 0x7f96cc0ad730 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: _finish_auth 0 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d4300640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96c8001690 con 0x7f96cc0ad730 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d4300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:05.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: dump: 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d4300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:05.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 1 -- 172.21.15.138:0/4281783335 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96c8001860 con 0x7f96cc0ad730 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d109a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d4300640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.130+0000 7f96d4300640 4 set_mon_vals no callback set 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.131+0000 7f96d4300640 10 monclient: shutdown 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.131+0000 7f96d4300640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.131+0000 7f96d4300640 1 -- 172.21.15.138:0/4281783335 >> v1:172.21.15.138:6789/0 conn(0x7f96cc0ad730 legacy=0x7f96cc14dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.131+0000 7f96d4300640 1 -- 172.21.15.138:0/4281783335 shutdown_connections 2024-03-29T11:21:05.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.131+0000 7f96d4300640 1 -- 172.21.15.138:0/4281783335 >> 172.21.15.138:0/4281783335 conn(0x7f96cc0b35e0 msgr2=0x7f96cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.131+0000 7f96d4300640 1 -- 172.21.15.138:0/4281783335 shutdown_connections 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.132+0000 7f96d4300640 1 -- 172.21.15.138:0/4281783335 wait complete. 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.133+0000 7f96d4300640 10 monclient: build_initial_monmap 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.133+0000 7f96d4300640 10 monclient: monmap: 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:05.113 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.133+0000 7f96d4300640 1 librados: starting msgr at 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.133+0000 7f96d4300640 1 librados: starting objecter 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.133+0000 7f96d4300640 1 Processor -- start 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.134+0000 7f96d4300640 1 -- start start 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.134+0000 7f96d4300640 1 librados: setting wanted keys 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.134+0000 7f96d4300640 1 librados: calling monclient init 2024-03-29T11:21:05.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.134+0000 7f96d4300640 10 monclient: init 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 10 monclient(hunting): picked mon.noname-a con 0x7f96cc0a96c0 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 10 monclient(hunting): picked mon.noname-b con 0x7f96cc0ad730 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 10 monclient(hunting): picked mon.noname-c con 0x7f96cc13ad10 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc13b1a0 con 0x7f96cc0a96c0 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc13e5c0 con 0x7f96cc0ad730 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc13ed10 con 0x7f96cc13ad10 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:05.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.135+0000 7f96d4300640 10 monclient(hunting): authenticate will time out at 886.444412s 2024-03-29T11:21:05.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.141+0000 7f96d189b640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f96cc0ad730 0x7f96cc13a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:37675/0 (socket says 172.21.15.138:37675) 2024-03-29T11:21:05.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.141+0000 7f96d189b640 1 -- 172.21.15.138:0/167329919 learned_addr learned my addr 172.21.15.138:0/167329919 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:05.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.141+0000 7f96c2ffd640 10 client.?.objecter ms_handle_connect 0x7f96cc0a96c0 2024-03-29T11:21:05.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.141+0000 7f96c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:05.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.141+0000 7f96c2ffd640 10 client.?.objecter ms_handle_connect 0x7f96cc13ad10 2024-03-29T11:21:05.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.141+0000 7f96c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:05.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 client.?.objecter ms_handle_connect 0x7f96cc0ad730 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124130624 0 0) 0x7f96cc13ed10 con 0x7f96cc13ad10 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b8004eb0 con 0x7f96cc13ad10 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802876771 0 0) 0x7f96cc13e5c0 con 0x7f96cc0ad730 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96cc13ed10 con 0x7f96cc0ad730 2024-03-29T11:21:05.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4186101506 0 0) 0x7f96b8004eb0 con 0x7f96cc13ad10 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96cc13e5c0 con 0x7f96cc13ad10 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96c8004940 con 0x7f96cc13ad10 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:05.125 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.142+0000 7f96c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2385496432 0 0) 0x7f96cc13e5c0 con 0x7f96cc13ad10 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 >> v1:172.21.15.152:6789/0 conn(0x7f96cc0ad730 legacy=0x7f96cc13a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 >> v1:172.21.15.138:6789/0 conn(0x7f96cc0a96c0 legacy=0x7f96cc13c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 1 monclient: found mon.c 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc13b350 con 0x7f96cc13ad10 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:05.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 10 monclient: _renew_subs 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc13d5e0 con 0x7f96cc13ad10 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 10 monclient: _renew_subs 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc13d3b0 con 0x7f96cc13ad10 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.143+0000 7f96d4300640 1 librados: init done 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.144+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96c8001b10 con 0x7f96cc13ad10 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.144+0000 7f96c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.144+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96c8004ef0 con 0x7f96cc13ad10 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.144+0000 7f96c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.144+0000 7f96c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.144+0000 7f96c2ffd640 10 monclient: dump: 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.144+0000 7f96c3fff640 4 set_mon_vals no callback set 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.145+0000 7f96d4300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.145+0000 7f96d4300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.145+0000 7f96d4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:05.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.145+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac005080 con 0x7f96cc13ad10 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.147+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f96c8005050 con 0x7f96cc13ad10 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.148+0000 7f96d209c640 10 monclient: get_auth_request con 0x7f96b806df50 auth_method 0 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.149+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f96c8084bc0 con 0x7f96cc13ad10 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.149+0000 7f96c2ffd640 10 client.4271.objecter ms_dispatch 0x7f96cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.149+0000 7f96c2ffd640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.149+0000 7f96c2ffd640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.149+0000 7f96c2ffd640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.149+0000 7f96c2ffd640 10 client.4271.objecter ms_handle_connect 0x7f96b806df50 2024-03-29T11:21:05.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.156+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96c80504a0 con 0x7f96cc13ad10 2024-03-29T11:21:05.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.156+0000 7f96c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.156+0000 7f96c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:05.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.482+0000 7f96d4300640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.482+0000 7f96d4300640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.482+0000 7f96d4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:05.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.482+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96ac005330 con 0x7f96cc13ad10 2024-03-29T11:21:05.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.483+0000 7f96c2ffd640 1 -- 172.21.15.138:0/167329919 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f96c80523d0 con 0x7f96cc13ad10 2024-03-29T11:21:05.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.483+0000 7f96c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.483+0000 7f96c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:05.460 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:05.461 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:05.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 10 librados: watch_flush enter 2024-03-29T11:21:05.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 10 librados: watch_flush exit 2024-03-29T11:21:05.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T11:21:05.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 10 client.4271.objecter successfully canceled tick 2024-03-29T11:21:05.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 >> v1:172.21.15.152:6816/3052585572 conn(0x7f96b806df50 legacy=0x7f96b8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 10 monclient: shutdown 2024-03-29T11:21:05.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:05.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 >> v1:172.21.15.138:6790/0 conn(0x7f96cc13ad10 legacy=0x7f96cc141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 shutdown_connections 2024-03-29T11:21:05.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 >> 172.21.15.138:0/167329919 conn(0x7f96cc0b35e0 msgr2=0x7f96cc0b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:05.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 shutdown_connections 2024-03-29T11:21:05.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.487+0000 7f96d4300640 1 -- 172.21.15.138:0/167329919 wait complete. 2024-03-29T11:21:05.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:05.488+0000 7f96d4300640 1 librados: shutdown 2024-03-29T11:21:05.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:12.286 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.462+0000 7fa0e27b6640 10 monclient: get_monmap_and_config 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.462+0000 7fa0e27b6640 10 monclient: build_initial_monmap 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.463+0000 7fa0e27b6640 10 monclient: monmap: 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:12.464452+0000 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:12.464452+0000 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:12.440 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:12.441 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.464+0000 7fa0e27b6640 1 Processor -- start 2024-03-29T11:21:12.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.464+0000 7fa0e27b6640 1 -- start start 2024-03-29T11:21:12.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.464+0000 7fa0e27b6640 10 monclient: init 2024-03-29T11:21:12.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:12.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:12.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 10 monclient(hunting): picked mon.noname-b con 0x7fa0dc0aa1b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:12.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 10 monclient(hunting): picked mon.noname-a con 0x7fa0dc13ba30 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:12.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 10 monclient(hunting): picked mon.noname-c con 0x7fa0dc13be90 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:12.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc0ac7c0 con 0x7fa0dc13ba30 2024-03-29T11:21:12.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc0adf60 con 0x7fa0dc13be90 2024-03-29T11:21:12.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc13e560 con 0x7fa0dc0aa1b0 2024-03-29T11:21:12.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:12.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0e27b6640 10 monclient(hunting): authenticate will time out at 893.774662s 2024-03-29T11:21:12.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0dbfff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fa0dc0aa1b0 0x7fa0dc0ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:42816/0 (socket says 172.21.15.138:42816) 2024-03-29T11:21:12.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.465+0000 7fa0dbfff640 1 -- 172.21.15.138:0/1150498448 learned_addr learned my addr 172.21.15.138:0/1150498448 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125997146 0 0) 0x7fa0dc0ac7c0 con 0x7fa0dc13ba30 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c8003520 con 0x7fa0dc13ba30 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 672539623 0 0) 0x7fa0c8003520 con 0x7fa0dc13ba30 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 >> v1:172.21.15.138:6790/0 conn(0x7fa0dc0aa1b0 legacy=0x7fa0dc0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 >> v1:172.21.15.152:6789/0 conn(0x7fa0dc13be90 legacy=0x7fa0dc149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 monclient: found mon.noname-a 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:21:12.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc0ab340 con 0x7fa0dc13ba30 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa0cc001690 con 0x7fa0dc13ba30 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.466+0000 7fa0daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 10 monclient: dump: 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.446 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0e27b6640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0e27b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0cc0017f0 con 0x7fa0dc13ba30 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0e27b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 1 -- 172.21.15.138:0/1150498448 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa0cc001c40 con 0x7fa0dc13ba30 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0e27b6640 4 set_mon_vals no callback set 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.467+0000 7fa0daffd640 10 monclient: dump: 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.468+0000 7fa0e27b6640 10 monclient: shutdown 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.468+0000 7fa0e27b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.468+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1150498448 >> v1:172.21.15.138:6789/0 conn(0x7fa0dc13ba30 legacy=0x7fa0dc0aac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.468+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1150498448 shutdown_connections 2024-03-29T11:21:12.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.468+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1150498448 >> 172.21.15.138:0/1150498448 conn(0x7fa0dc0b3c50 msgr2=0x7fa0dc0b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:12.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.468+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1150498448 shutdown_connections 2024-03-29T11:21:12.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.468+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1150498448 wait complete. 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.469+0000 7fa0e27b6640 10 monclient: build_initial_monmap 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.469+0000 7fa0e27b6640 10 monclient: monmap: 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.454 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.469+0000 7fa0e27b6640 1 librados: starting msgr at 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.469+0000 7fa0e27b6640 1 librados: starting objecter 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.469+0000 7fa0e27b6640 1 Processor -- start 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.470+0000 7fa0e27b6640 1 -- start start 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.470+0000 7fa0e27b6640 1 librados: setting wanted keys 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.470+0000 7fa0e27b6640 1 librados: calling monclient init 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.470+0000 7fa0e27b6640 10 monclient: init 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.470+0000 7fa0e27b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.470+0000 7fa0e27b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 10 monclient(hunting): picked mon.noname-c con 0x7fa0dc0aa1b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 10 monclient(hunting): picked mon.noname-b con 0x7fa0dc13ba30 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:12.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 10 monclient(hunting): picked mon.noname-a con 0x7fa0dc13be90 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc06fc40 con 0x7fa0dc13be90 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc06ff20 con 0x7fa0dc13ba30 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc06e760 con 0x7fa0dc0aa1b0 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0e27b6640 10 monclient(hunting): authenticate will time out at 893.780123s 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0db7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fa0dc13ba30 0x7fa0dc1dbd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:40148/0 (socket says 172.21.15.138:40148) 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0db7fe640 1 -- 172.21.15.138:0/1520201680 learned_addr learned my addr 172.21.15.138:0/1520201680 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0d8ff9640 10 client.?.objecter ms_handle_connect 0x7fa0dc0aa1b0 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.471+0000 7fa0d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 client.?.objecter ms_handle_connect 0x7fa0dc13be90 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 client.?.objecter ms_handle_connect 0x7fa0dc13ba30 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372650130 0 0) 0x7fa0dc06fc40 con 0x7fa0dc13be90 2024-03-29T11:21:12.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0bc004eb0 con 0x7fa0dc13be90 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.472+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282588292 0 0) 0x7fa0dc06ff20 con 0x7fa0dc13ba30 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc06fc40 con 0x7fa0dc13ba30 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883252782 0 0) 0x7fa0dc06e760 con 0x7fa0dc0aa1b0 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc06ff20 con 0x7fa0dc0aa1b0 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2043606959 0 0) 0x7fa0dc06fc40 con 0x7fa0dc13ba30 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0dc06e760 con 0x7fa0dc13ba30 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa0cc001b40 con 0x7fa0dc13ba30 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.458 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459459685 0 0) 0x7fa0bc004eb0 con 0x7fa0dc13be90 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0dc06fc40 con 0x7fa0dc13be90 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa0c4001750 con 0x7fa0dc13be90 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.473+0000 7fa0d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.459 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390752336 0 0) 0x7fa0dc06ff20 con 0x7fa0dc0aa1b0 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0bc004eb0 con 0x7fa0dc0aa1b0 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa0d4001ae0 con 0x7fa0dc0aa1b0 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:12.460 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4055736111 0 0) 0x7fa0dc06e760 con 0x7fa0dc13ba30 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 >> v1:172.21.15.138:6790/0 conn(0x7fa0dc0aa1b0 legacy=0x7fa0dc1db600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 >> v1:172.21.15.138:6789/0 conn(0x7fa0dc13be90 legacy=0x7fa0dc1e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 monclient: found mon.b 2024-03-29T11:21:12.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc131740 con 0x7fa0dc13ba30 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 10 monclient: _renew_subs 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0dc1319f0 con 0x7fa0dc13ba30 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 10 monclient: _renew_subs 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc06e240 con 0x7fa0dc13ba30 2024-03-29T11:21:12.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.474+0000 7fa0e27b6640 1 librados: init done 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.475+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0cc0038a0 con 0x7fa0dc13ba30 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.475+0000 7fa0d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.475+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa0cc001ca0 con 0x7fa0dc13ba30 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.475+0000 7fa0da7fc640 4 set_mon_vals no callback set 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.476+0000 7fa0e27b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.476+0000 7fa0e27b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.476+0000 7fa0e27b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.476+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c0005080 con 0x7fa0dc13ba30 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.476+0000 7fa0d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.476+0000 7fa0d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.476+0000 7fa0d8ff9640 10 monclient: dump: 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:12.463 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:12.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.477+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fa0cc003cb0 con 0x7fa0dc13ba30 2024-03-29T11:21:12.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.478+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fa0cc084a70 con 0x7fa0dc13ba30 2024-03-29T11:21:12.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.478+0000 7fa0d8ff9640 10 client.4264.objecter ms_dispatch 0x7fa0dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:12.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.478+0000 7fa0d8ff9640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:12.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.478+0000 7fa0d8ff9640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:12.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.478+0000 7fa0d8ff9640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:21:12.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.478+0000 7fa0dbfff640 10 monclient: get_auth_request con 0x7fa0bc06e120 auth_method 0 2024-03-29T11:21:12.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.479+0000 7fa0d8ff9640 10 client.4264.objecter ms_handle_connect 0x7fa0bc06e120 2024-03-29T11:21:12.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.484+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0cc050350 con 0x7fa0dc13ba30 2024-03-29T11:21:12.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.484+0000 7fa0d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.484+0000 7fa0d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:12.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.868+0000 7fa0e27b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.868+0000 7fa0e27b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.868+0000 7fa0e27b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:12.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.868+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0c0005330 con 0x7fa0dc13ba30 2024-03-29T11:21:12.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.869+0000 7fa0d8ff9640 1 -- 172.21.15.138:0/1520201680 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fa0cc052280 con 0x7fa0dc13ba30 2024-03-29T11:21:12.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.869+0000 7fa0d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.869+0000 7fa0d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:12.847 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:12.847 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:12.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.874+0000 7fa0e27b6640 10 librados: watch_flush enter 2024-03-29T11:21:12.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.874+0000 7fa0e27b6640 10 librados: watch_flush exit 2024-03-29T11:21:12.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.874+0000 7fa0e27b6640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:21:12.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.874+0000 7fa0e27b6640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:21:12.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.874+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 >> v1:172.21.15.152:6816/3052585572 conn(0x7fa0bc06e120 legacy=0x7fa0bc0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.874+0000 7fa0e27b6640 10 monclient: shutdown 2024-03-29T11:21:12.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.875+0000 7fa0e27b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:12.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.875+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 >> v1:172.21.15.152:6789/0 conn(0x7fa0dc13ba30 legacy=0x7fa0dc1dbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.875+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 shutdown_connections 2024-03-29T11:21:12.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.875+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 >> 172.21.15.138:0/1520201680 conn(0x7fa0dc0b3c50 msgr2=0x7fa0dc0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:12.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.875+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 shutdown_connections 2024-03-29T11:21:12.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.875+0000 7fa0e27b6640 1 -- 172.21.15.138:0/1520201680 wait complete. 2024-03-29T11:21:12.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:12.876+0000 7fa0e27b6640 1 librados: shutdown 2024-03-29T11:21:12.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:19.675 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:19.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.833+0000 7f2eea372640 10 monclient: get_monmap_and_config 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.833+0000 7f2eea372640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.833+0000 7f2eea372640 10 monclient: monmap: 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:19.834345+0000 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:19.834345+0000 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:19.810 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.834+0000 7f2eea372640 1 Processor -- start 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.834+0000 7f2eea372640 1 -- start start 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.834+0000 7f2eea372640 10 monclient: init 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee406c250 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee40ae650 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:19.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee413f8c0 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:19.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40adf50 con 0x7f2ee413f8c0 2024-03-29T11:21:19.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40ae230 con 0x7f2ee40ae650 2024-03-29T11:21:19.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee413fd50 con 0x7f2ee406c250 2024-03-29T11:21:19.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2eea372640 10 monclient(hunting): authenticate will time out at 901.144364s 2024-03-29T11:21:19.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2ee37fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f2ee40ae650 0x7f2ee413f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:57966/0 (socket says 172.21.15.138:57966) 2024-03-29T11:21:19.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.835+0000 7f2ee37fe640 1 -- 172.21.15.138:0/1276840577 learned_addr learned my addr 172.21.15.138:0/1276840577 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905698623 0 0) 0x7f2ee413fd50 con 0x7f2ee406c250 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ecc003520 con 0x7f2ee406c250 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043003098 0 0) 0x7f2ee40ae230 con 0x7f2ee40ae650 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.817 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.817 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:21:19.817 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.839+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ecc0048a0 con 0x7f2ee40ae650 2024-03-29T11:21:19.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805841234 0 0) 0x7f2ecc0048a0 con 0x7f2ee40ae650 2024-03-29T11:21:19.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 >> v1:172.21.15.138:6790/0 conn(0x7f2ee406c250 legacy=0x7f2ee41427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 >> v1:172.21.15.138:6789/0 conn(0x7f2ee413f8c0 legacy=0x7f2ee414db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee406db20 con 0x7f2ee40ae650 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2ed8001670 con 0x7f2ee40ae650 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: dump: 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ed80017d0 con 0x7f2ee40ae650 2024-03-29T11:21:19.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 1 -- 172.21.15.138:0/1276840577 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2ed8001be0 con 0x7f2ee40ae650 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2ee2ffd640 10 monclient: dump: 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:19.821 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2eea372640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2eea372640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.840+0000 7f2eea372640 4 set_mon_vals no callback set 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.841+0000 7f2eea372640 10 monclient: shutdown 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.841+0000 7f2eea372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.841+0000 7f2eea372640 1 -- 172.21.15.138:0/1276840577 >> v1:172.21.15.152:6789/0 conn(0x7f2ee40ae650 legacy=0x7f2ee413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.841+0000 7f2eea372640 1 -- 172.21.15.138:0/1276840577 shutdown_connections 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.841+0000 7f2eea372640 1 -- 172.21.15.138:0/1276840577 >> 172.21.15.138:0/1276840577 conn(0x7f2ee4067610 msgr2=0x7f2ee40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.842+0000 7f2eea372640 1 -- 172.21.15.138:0/1276840577 shutdown_connections 2024-03-29T11:21:19.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.842+0000 7f2eea372640 1 -- 172.21.15.138:0/1276840577 wait complete. 2024-03-29T11:21:19.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.843+0000 7f2eea372640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.843+0000 7f2eea372640 10 monclient: monmap: 2024-03-29T11:21:19.831 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:19.831 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.843+0000 7f2eea372640 1 librados: starting msgr at 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.843+0000 7f2eea372640 1 librados: starting objecter 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.843+0000 7f2eea372640 1 Processor -- start 2024-03-29T11:21:19.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.844+0000 7f2eea372640 1 -- start start 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.844+0000 7f2eea372640 1 librados: setting wanted keys 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.844+0000 7f2eea372640 1 librados: calling monclient init 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.844+0000 7f2eea372640 10 monclient: init 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee406c250 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee40ae650 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee413f8c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee4139df0 con 0x7f2ee40ae650 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee4145e80 con 0x7f2ee413f8c0 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40ad640 con 0x7f2ee406c250 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2eea372640 10 monclient(hunting): authenticate will time out at 901.154498s 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2ee890f640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f2ee413f8c0 0x7f2ee41396e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:58134/0 (socket says 172.21.15.138:58134) 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2ee890f640 1 -- 172.21.15.138:0/615339702 learned_addr learned my addr 172.21.15.138:0/615339702 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ee413f8c0 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.845+0000 7f2ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589081736 0 0) 0x7f2ee4145e80 con 0x7f2ee413f8c0 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed0004eb0 con 0x7f2ee413f8c0 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ee406c250 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3866591377 0 0) 0x7f2ed0004eb0 con 0x7f2ee413f8c0 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ee4145e80 con 0x7f2ee413f8c0 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2edc001c70 con 0x7f2ee413f8c0 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:19.835 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3206175617 0 0) 0x7f2ee4145e80 con 0x7f2ee413f8c0 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 >> v1:172.21.15.138:6790/0 conn(0x7f2ee406c250 legacy=0x7f2ee4143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 >> v1:172.21.15.138:6789/0 conn(0x7f2ee40ae650 legacy=0x7f2ee4138fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 monclient: found mon.b 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee41463a0 con 0x7f2ee413f8c0 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 10 monclient: _renew_subs 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee4072880 con 0x7f2ee413f8c0 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 10 monclient: _renew_subs 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee4144690 con 0x7f2ee413f8c0 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2eea372640 1 librados: init done 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2edc002070 con 0x7f2ee413f8c0 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2edc003de0 con 0x7f2ee413f8c0 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee0ff9640 10 monclient: dump: 2024-03-29T11:21:19.837 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.846+0000 7f2ee1ffb640 4 set_mon_vals no callback set 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.848+0000 7f2eea372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.848+0000 7f2eea372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.848+0000 7f2eea372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.848+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec4005080 con 0x7f2ee413f8c0 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.849+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f2edc003f40 con 0x7f2ee413f8c0 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.849+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f2edc084e70 con 0x7f2ee413f8c0 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.849+0000 7f2ee0ff9640 10 client.4270.objecter ms_dispatch 0x7f2ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.849+0000 7f2ee0ff9640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.849+0000 7f2ee0ff9640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.849+0000 7f2ee0ff9640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.849+0000 7f2ee3fff640 10 monclient: get_auth_request con 0x7f2ed006dbb0 auth_method 0 2024-03-29T11:21:19.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.850+0000 7f2ee0ff9640 10 client.4270.objecter ms_handle_connect 0x7f2ed006dbb0 2024-03-29T11:21:19.840 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.859+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2edc050750 con 0x7f2ee413f8c0 2024-03-29T11:21:19.840 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.859+0000 7f2ee0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.840 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:19.859+0000 7f2ee0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:20.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.168+0000 7f2eea372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:20.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.168+0000 7f2eea372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:20.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.168+0000 7f2eea372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:20.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.168+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ec4005330 con 0x7f2ee413f8c0 2024-03-29T11:21:20.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.169+0000 7f2ee0ff9640 1 -- 172.21.15.138:0/615339702 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f2edc052680 con 0x7f2ee413f8c0 2024-03-29T11:21:20.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.169+0000 7f2ee0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:20.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.169+0000 7f2ee0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:20.146 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:20.146 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.172+0000 7f2eea372640 10 librados: watch_flush enter 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.172+0000 7f2eea372640 10 librados: watch_flush exit 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.172+0000 7f2eea372640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.172+0000 7f2eea372640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.172+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 >> v1:172.21.15.152:6816/3052585572 conn(0x7f2ed006dbb0 legacy=0x7f2ed0070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.172+0000 7f2eea372640 10 monclient: shutdown 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.172+0000 7f2eea372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.173+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 >> v1:172.21.15.152:6789/0 conn(0x7f2ee413f8c0 legacy=0x7f2ee41396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.173+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 shutdown_connections 2024-03-29T11:21:20.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.173+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 >> 172.21.15.138:0/615339702 conn(0x7f2ee4067610 msgr2=0x7f2ee413ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:20.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.173+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 shutdown_connections 2024-03-29T11:21:20.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.173+0000 7f2eea372640 1 -- 172.21.15.138:0/615339702 wait complete. 2024-03-29T11:21:20.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:20.173+0000 7f2eea372640 1 librados: shutdown 2024-03-29T11:21:20.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:26.971 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:27.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.148+0000 7ff15cc4a640 10 monclient: get_monmap_and_config 2024-03-29T11:21:27.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.148+0000 7ff15cc4a640 10 monclient: build_initial_monmap 2024-03-29T11:21:27.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.148+0000 7ff15cc4a640 10 monclient: monmap: 2024-03-29T11:21:27.125 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:27.125 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:27.149313+0000 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:27.149313+0000 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:27.126 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:27.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.150+0000 7ff15cc4a640 1 Processor -- start 2024-03-29T11:21:27.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.150+0000 7ff15cc4a640 1 -- start start 2024-03-29T11:21:27.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.150+0000 7ff15cc4a640 10 monclient: init 2024-03-29T11:21:27.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:27.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:27.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1580b4330 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:27.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1581470b0 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:27.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1581439b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:27.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15806f2a0 con 0x7ff1581470b0 2024-03-29T11:21:27.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff158146b90 con 0x7ff1581439b0 2024-03-29T11:21:27.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1580b2ca0 con 0x7ff1580b4330 2024-03-29T11:21:27.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:27.129 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.152+0000 7ff15cc4a640 10 monclient(hunting): authenticate will time out at 908.461330s 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff155d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ff1581470b0 0x7ff1580b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34415/0 (socket says 172.21.15.138:34415) 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff155d9b640 1 -- 172.21.15.138:0/1963557145 learned_addr learned my addr 172.21.15.138:0/1963557145 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15659c640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7ff1580b4330 0x7ff1580b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:42441/0 (socket says 172.21.15.138:42441) 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278724280 0 0) 0x7ff15806f2a0 con 0x7ff1581470b0 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:21:27.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff140003520 con 0x7ff1581470b0 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867104334 0 0) 0x7ff1580b2ca0 con 0x7ff1580b4330 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff15806f2a0 con 0x7ff1580b4330 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1344001431 0 0) 0x7ff15806f2a0 con 0x7ff1580b4330 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 >> v1:172.21.15.152:6789/0 conn(0x7ff1581439b0 legacy=0x7ff158151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 >> v1:172.21.15.138:6789/0 conn(0x7ff1581470b0 legacy=0x7ff1580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.155+0000 7ff15559a640 1 monclient: found mon.noname-b 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:27.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15806e0c0 con 0x7ff1580b4330 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: _finish_auth 0 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff148001670 con 0x7ff1580b4330 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: dump: 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.135 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15cc4a640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15cc4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1480017d0 con 0x7ff1580b4330 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 1 -- 172.21.15.138:0/1963557145 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff148001ba0 con 0x7ff1580b4330 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15559a640 10 monclient: dump: 2024-03-29T11:21:27.136 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15cc4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:27.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.156+0000 7ff15cc4a640 4 set_mon_vals no callback set 2024-03-29T11:21:27.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.157+0000 7ff15cc4a640 10 monclient: shutdown 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.157+0000 7ff15cc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.157+0000 7ff15cc4a640 1 -- 172.21.15.138:0/1963557145 >> v1:172.21.15.138:6790/0 conn(0x7ff1580b4330 legacy=0x7ff1580b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.157+0000 7ff15cc4a640 1 -- 172.21.15.138:0/1963557145 shutdown_connections 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.157+0000 7ff15cc4a640 1 -- 172.21.15.138:0/1963557145 >> 172.21.15.138:0/1963557145 conn(0x7ff158066a00 msgr2=0x7ff158066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.157+0000 7ff15cc4a640 1 -- 172.21.15.138:0/1963557145 shutdown_connections 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.158+0000 7ff15cc4a640 1 -- 172.21.15.138:0/1963557145 wait complete. 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.159+0000 7ff15cc4a640 10 monclient: build_initial_monmap 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.159+0000 7ff15cc4a640 10 monclient: monmap: 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:27.141 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.159+0000 7ff15cc4a640 1 librados: starting msgr at 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.159+0000 7ff15cc4a640 1 librados: starting objecter 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 1 Processor -- start 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 1 -- start start 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 1 librados: setting wanted keys 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 1 librados: calling monclient init 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient: init 2024-03-29T11:21:27.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1580b4330 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1581439b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1581470b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff158143040 con 0x7ff1580b4330 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff158142100 con 0x7ff1581439b0 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1580a94e0 con 0x7ff1581470b0 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.160+0000 7ff15cc4a640 10 monclient(hunting): authenticate will time out at 908.469921s 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.161+0000 7ff155d9b640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ff1581439b0 0x7ff1581d2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:58066/0 (socket says 172.21.15.138:58066) 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.161+0000 7ff155d9b640 1 -- 172.21.15.138:0/3521940531 learned_addr learned my addr 172.21.15.138:0/3521940531 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 10 client.?.objecter ms_handle_connect 0x7ff1581439b0 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110349742 0 0) 0x7ff158142100 con 0x7ff1581439b0 2024-03-29T11:21:27.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:27.144 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:27.144 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:27.144 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:21:27.144 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.163+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138004eb0 con 0x7ff1581439b0 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.164+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262003011 0 0) 0x7ff138004eb0 con 0x7ff1581439b0 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.165+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff158142100 con 0x7ff1581439b0 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.165+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff14c001950 con 0x7ff1581439b0 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.165+0000 7ff146ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.165+0000 7ff146ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.165+0000 7ff146ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.147 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.165+0000 7ff146ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 10 client.?.objecter ms_handle_connect 0x7ff1581470b0 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038459299 0 0) 0x7ff1580a94e0 con 0x7ff1581470b0 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.166+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138004eb0 con 0x7ff1581470b0 2024-03-29T11:21:27.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.167+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3048852422 0 0) 0x7ff158142100 con 0x7ff1581439b0 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.167+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 >> v1:172.21.15.138:6790/0 conn(0x7ff1581470b0 legacy=0x7ff1581de4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.167+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 >> v1:172.21.15.138:6789/0 conn(0x7ff1580b4330 legacy=0x7ff15813ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.168+0000 7ff146ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.168+0000 7ff146ffd640 1 monclient: found mon.b 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff146ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff146ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1580a9670 con 0x7ff1581439b0 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff146ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff146ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff146ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff15cc4a640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff15cc4a640 10 monclient: _renew_subs 2024-03-29T11:21:27.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff15cc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff158070aa0 con 0x7ff1581439b0 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff15cc4a640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff15cc4a640 10 monclient: _renew_subs 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.169+0000 7ff15cc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.170+0000 7ff146ffd640 10 client.4273.objecter ms_handle_connect 0x7ff1580b4330 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.170+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15813b380 con 0x7ff1581439b0 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.170+0000 7ff15cc4a640 1 librados: init done 2024-03-29T11:21:27.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.170+0000 7ff146ffd640 10 client.4273.objecter resend_mon_ops 2024-03-29T11:21:27.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff14c0047e0 con 0x7ff1581439b0 2024-03-29T11:21:27.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff146ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:27.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff14c004bb0 con 0x7ff1581439b0 2024-03-29T11:21:27.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff146ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff146ffd640 10 monclient: dump: 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:27.153 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff14c004dd0 con 0x7ff1581439b0 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff15cc4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff15cc4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff15cc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff158075f10 con 0x7ff1581439b0 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.173+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ff14c084c40 con 0x7ff1581439b0 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.173+0000 7ff146ffd640 10 client.4273.objecter ms_dispatch 0x7ff158000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.173+0000 7ff146ffd640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.173+0000 7ff146ffd640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.173+0000 7ff146ffd640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.172+0000 7ff154d99640 4 set_mon_vals no callback set 2024-03-29T11:21:27.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.174+0000 7ff15659c640 10 monclient: get_auth_request con 0x7ff13806dd90 auth_method 0 2024-03-29T11:21:27.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.175+0000 7ff146ffd640 10 client.4273.objecter ms_handle_connect 0x7ff13806dd90 2024-03-29T11:21:27.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.181+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff14c0505a0 con 0x7ff1581439b0 2024-03-29T11:21:27.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.181+0000 7ff146ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.181+0000 7ff146ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:27.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.504+0000 7ff15cc4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.504+0000 7ff15cc4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.504+0000 7ff15cc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:21:27.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.504+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff158074f10 con 0x7ff1581439b0 2024-03-29T11:21:27.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.509+0000 7ff146ffd640 1 -- 172.21.15.138:0/3521940531 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ff14c0524d0 con 0x7ff1581439b0 2024-03-29T11:21:27.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.509+0000 7ff146ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.509+0000 7ff146ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:27.486 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:27.487 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.512+0000 7ff15cc4a640 10 librados: watch_flush enter 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.512+0000 7ff15cc4a640 10 librados: watch_flush exit 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.512+0000 7ff15cc4a640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.513+0000 7ff15cc4a640 10 client.4273.objecter successfully canceled tick 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.513+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff13806dd90 legacy=0x7ff138070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.513+0000 7ff15cc4a640 10 monclient: shutdown 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.513+0000 7ff15cc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:27.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.513+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 >> v1:172.21.15.152:6789/0 conn(0x7ff1581439b0 legacy=0x7ff1581d2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.514+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 shutdown_connections 2024-03-29T11:21:27.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.515+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 >> 172.21.15.138:0/3521940531 conn(0x7ff158066a00 msgr2=0x7ff158145c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:27.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.516+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 shutdown_connections 2024-03-29T11:21:27.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.516+0000 7ff15cc4a640 1 -- 172.21.15.138:0/3521940531 wait complete. 2024-03-29T11:21:27.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:27.519+0000 7ff15cc4a640 1 librados: shutdown 2024-03-29T11:21:27.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:34.321 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:34.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.500+0000 7f5b88a5b640 10 monclient: get_monmap_and_config 2024-03-29T11:21:34.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.500+0000 7f5b88a5b640 10 monclient: build_initial_monmap 2024-03-29T11:21:34.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.500+0000 7f5b88a5b640 10 monclient: monmap: 2024-03-29T11:21:34.476 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:34.476 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:34.500693+0000 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:34.500693+0000 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:34.477 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:34.480 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.505+0000 7f5b88a5b640 1 Processor -- start 2024-03-29T11:21:34.480 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.505+0000 7f5b88a5b640 1 -- start start 2024-03-29T11:21:34.481 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.505+0000 7f5b88a5b640 10 monclient: init 2024-03-29T11:21:34.481 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:34.481 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:34.481 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 10 monclient(hunting): picked mon.noname-a con 0x7f5b840b2800 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:34.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 10 monclient(hunting): picked mon.noname-b con 0x7f5b840b3220 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:34.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 10 monclient(hunting): picked mon.noname-c con 0x7f5b841439c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:34.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b840b3840 con 0x7f5b840b2800 2024-03-29T11:21:34.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8406f230 con 0x7f5b841439c0 2024-03-29T11:21:34.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8406ef50 con 0x7f5b840b3220 2024-03-29T11:21:34.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:34.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.506+0000 7f5b88a5b640 10 monclient(hunting): authenticate will time out at 915.815010s 2024-03-29T11:21:34.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.509+0000 7f5b8259c640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f5b840b3220 0x7f5b84146710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:58529/0 (socket says 172.21.15.138:58529) 2024-03-29T11:21:34.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.509+0000 7f5b8259c640 1 -- 172.21.15.138:0/3519324723 learned_addr learned my addr 172.21.15.138:0/3519324723 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:34.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.512+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131416848 0 0) 0x7f5b8406ef50 con 0x7f5b840b3220 2024-03-29T11:21:34.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.512+0000 7f5b81d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.512+0000 7f5b81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.512+0000 7f5b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.512+0000 7f5b81d9b640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:21:34.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.512+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b6c003520 con 0x7f5b840b3220 2024-03-29T11:21:34.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415996017 0 0) 0x7f5b840b3840 con 0x7f5b840b2800 2024-03-29T11:21:34.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:21:34.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b8406ef50 con 0x7f5b840b2800 2024-03-29T11:21:34.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240747345 0 0) 0x7f5b8406f230 con 0x7f5b841439c0 2024-03-29T11:21:34.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b840b3840 con 0x7f5b841439c0 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830222291 0 0) 0x7f5b8406ef50 con 0x7f5b840b2800 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.513+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 >> v1:172.21.15.138:6790/0 conn(0x7f5b840b3220 legacy=0x7f5b84146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 >> v1:172.21.15.152:6789/0 conn(0x7f5b841439c0 legacy=0x7f5b84151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 1 monclient: found mon.noname-a 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b84071d30 con 0x7f5b840b2800 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:34.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5b78001670 con 0x7f5b840b2800 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: dump: 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.492 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.493 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:34.493 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:34.493 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:34.493 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:34.493 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:34.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b88a5b640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:21:34.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b88a5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:34.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b780017d0 con 0x7f5b840b2800 2024-03-29T11:21:34.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:34.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 1 -- 172.21.15.138:0/3519324723 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5b78001be0 con 0x7f5b840b2800 2024-03-29T11:21:34.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b81d9b640 10 monclient: dump: 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:34.496 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b88a5b640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.514+0000 7f5b88a5b640 4 set_mon_vals no callback set 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.515+0000 7f5b88a5b640 10 monclient: shutdown 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.515+0000 7f5b88a5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.515+0000 7f5b88a5b640 1 -- 172.21.15.138:0/3519324723 >> v1:172.21.15.138:6789/0 conn(0x7f5b840b2800 legacy=0x7f5b841432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.515+0000 7f5b88a5b640 1 -- 172.21.15.138:0/3519324723 shutdown_connections 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.515+0000 7f5b88a5b640 1 -- 172.21.15.138:0/3519324723 >> 172.21.15.138:0/3519324723 conn(0x7f5b84066a00 msgr2=0x7f5b84066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.518+0000 7f5b88a5b640 1 -- 172.21.15.138:0/3519324723 shutdown_connections 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.518+0000 7f5b88a5b640 1 -- 172.21.15.138:0/3519324723 wait complete. 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.519+0000 7f5b88a5b640 10 monclient: build_initial_monmap 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.519+0000 7f5b88a5b640 10 monclient: monmap: 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:34.498 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:34.499 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.520+0000 7f5b88a5b640 1 librados: starting msgr at 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.520+0000 7f5b88a5b640 1 librados: starting objecter 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.520+0000 7f5b88a5b640 1 Processor -- start 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.521+0000 7f5b88a5b640 1 -- start start 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.521+0000 7f5b88a5b640 1 librados: setting wanted keys 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.521+0000 7f5b88a5b640 1 librados: calling monclient init 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.521+0000 7f5b88a5b640 10 monclient: init 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.522+0000 7f5b88a5b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:34.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.522+0000 7f5b88a5b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:34.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.525+0000 7f5b88a5b640 10 monclient(hunting): picked mon.noname-a con 0x7f5b840b2800 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:34.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.528+0000 7f5b8259c640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f5b840b3220 0x7f5b840b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:56949/0 (socket says 172.21.15.138:56949) 2024-03-29T11:21:34.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.528+0000 7f5b8259c640 1 -- 172.21.15.138:0/375753454 learned_addr learned my addr 172.21.15.138:0/375753454 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:34.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.528+0000 7f5b88a5b640 10 monclient(hunting): picked mon.noname-b con 0x7f5b840b3220 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:34.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.528+0000 7f5b88a5b640 10 monclient(hunting): picked mon.noname-c con 0x7f5b841439c0 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:34.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b88a5b640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b84149e00 con 0x7f5b840b2800 2024-03-29T11:21:34.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b88a5b640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8414a030 con 0x7f5b840b3220 2024-03-29T11:21:34.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b88a5b640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8413c870 con 0x7f5b841439c0 2024-03-29T11:21:34.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b88a5b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:34.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b88a5b640 10 monclient(hunting): authenticate will time out at 915.840552s 2024-03-29T11:21:34.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b6b7fe640 10 client.?.objecter ms_handle_connect 0x7f5b840b3220 2024-03-29T11:21:34.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.532+0000 7f5b6b7fe640 10 client.?.objecter ms_handle_connect 0x7f5b841439c0 2024-03-29T11:21:34.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 client.?.objecter ms_handle_connect 0x7f5b840b2800 2024-03-29T11:21:34.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848366055 0 0) 0x7f5b84149e00 con 0x7f5b840b2800 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b70004eb0 con 0x7f5b840b2800 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739069921 0 0) 0x7f5b8414a030 con 0x7f5b840b3220 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.533+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b84149e00 con 0x7f5b840b3220 2024-03-29T11:21:34.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879941474 0 0) 0x7f5b8413c870 con 0x7f5b841439c0 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b8414a030 con 0x7f5b841439c0 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 360990994 0 0) 0x7f5b70004eb0 con 0x7f5b840b2800 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b8413c870 con 0x7f5b840b2800 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807440127 0 0) 0x7f5b8414a030 con 0x7f5b841439c0 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b70004eb0 con 0x7f5b841439c0 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5b78001ae0 con 0x7f5b840b2800 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:34.511 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5b7c001b50 con 0x7f5b841439c0 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:34.512 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:34.513 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 977905302 0 0) 0x7f5b8413c870 con 0x7f5b840b2800 2024-03-29T11:21:34.514 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 >> v1:172.21.15.138:6790/0 conn(0x7f5b841439c0 legacy=0x7f5b840b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.514 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.534+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 >> v1:172.21.15.152:6789/0 conn(0x7f5b840b3220 legacy=0x7f5b840b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.514 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:34.514 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 1 monclient: found mon.a 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8413c300 con 0x7f5b840b2800 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b780038a0 con 0x7f5b840b2800 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5b78003bb0 con 0x7f5b840b2800 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.535+0000 7f5b6b7fe640 10 monclient: dump: 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:34.516 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 10 monclient: _renew_subs 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8413c5b0 con 0x7f5b840b2800 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b8159a640 4 set_mon_vals no callback set 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 10 monclient: _renew_subs 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:34.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b84148090 con 0x7f5b840b2800 2024-03-29T11:21:34.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.538+0000 7f5b88a5b640 1 librados: init done 2024-03-29T11:21:34.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.539+0000 7f5b697fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.539+0000 7f5b697fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.539+0000 7f5b697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:34.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.539+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c005080 con 0x7f5b840b2800 2024-03-29T11:21:34.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.540+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f5b78019b80 con 0x7f5b840b2800 2024-03-29T11:21:34.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.541+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f5b78083900 con 0x7f5b840b2800 2024-03-29T11:21:34.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.541+0000 7f5b6b7fe640 10 client.4293.objecter ms_dispatch 0x7f5b84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:34.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.541+0000 7f5b6b7fe640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:34.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.541+0000 7f5b6b7fe640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:34.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.541+0000 7f5b6b7fe640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T11:21:34.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.545+0000 7f5b8259c640 10 monclient: get_auth_request con 0x7f5b7006e100 auth_method 0 2024-03-29T11:21:34.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.546+0000 7f5b6b7fe640 10 client.4293.objecter ms_handle_connect 0x7f5b7006e100 2024-03-29T11:21:34.525 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.549+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b7804f320 con 0x7f5b840b2800 2024-03-29T11:21:34.525 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.549+0000 7f5b6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.525 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.549+0000 7f5b6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:34.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.871+0000 7f5b697fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.871+0000 7f5b697fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.871+0000 7f5b697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:34.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.872+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b5c005330 con 0x7f5b840b2800 2024-03-29T11:21:34.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.872+0000 7f5b6b7fe640 1 -- 172.21.15.138:0/375753454 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f5b78051250 con 0x7f5b840b2800 2024-03-29T11:21:34.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.872+0000 7f5b6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.872+0000 7f5b6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:34.849 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:34.850 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:34.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.876+0000 7f5b697fa640 10 librados: watch_flush enter 2024-03-29T11:21:34.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.876+0000 7f5b697fa640 10 librados: watch_flush exit 2024-03-29T11:21:34.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.876+0000 7f5b697fa640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T11:21:34.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.876+0000 7f5b697fa640 10 client.4293.objecter successfully canceled tick 2024-03-29T11:21:34.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.877+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 >> v1:172.21.15.152:6816/3052585572 conn(0x7f5b7006e100 legacy=0x7f5b700705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.877+0000 7f5b697fa640 10 monclient: shutdown 2024-03-29T11:21:34.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.877+0000 7f5b697fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.877+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 >> v1:172.21.15.138:6789/0 conn(0x7f5b840b2800 legacy=0x7f5b8413bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.879+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 shutdown_connections 2024-03-29T11:21:34.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.879+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 >> 172.21.15.138:0/375753454 conn(0x7f5b84066a00 msgr2=0x7f5b840b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.879+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 shutdown_connections 2024-03-29T11:21:34.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.879+0000 7f5b697fa640 1 -- 172.21.15.138:0/375753454 wait complete. 2024-03-29T11:21:34.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:34.879+0000 7f5b697fa640 1 librados: shutdown 2024-03-29T11:21:34.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:41.679 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:41.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.853+0000 7fdb8d650640 10 monclient: get_monmap_and_config 2024-03-29T11:21:41.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.853+0000 7fdb8d650640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.853+0000 7fdb8d650640 10 monclient: monmap: 2024-03-29T11:21:41.830 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:41.830 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:41.830 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:41.853933+0000 2024-03-29T11:21:41.830 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:41.853933+0000 2024-03-29T11:21:41.831 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.831 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:41.831 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:41.831 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:41.831 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:41.831 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:41.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.856+0000 7fdb8d650640 1 Processor -- start 2024-03-29T11:21:41.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.856+0000 7fdb8d650640 1 -- start start 2024-03-29T11:21:41.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.856+0000 7fdb8d650640 10 monclient: init 2024-03-29T11:21:41.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 10 monclient(hunting): picked mon.noname-c con 0x7fdb880b4100 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 10 monclient(hunting): picked mon.noname-b con 0x7fdb8813ee90 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 10 monclient(hunting): picked mon.noname-a con 0x7fdb880af900 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880afd10 con 0x7fdb880af900 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880af6f0 con 0x7fdb880b4100 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880aaf60 con 0x7fdb8813ee90 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.857+0000 7fdb8d650640 10 monclient(hunting): authenticate will time out at 923.165405s 2024-03-29T11:21:41.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.863+0000 7fdb867fc640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fdb8813ee90 0x7fdb880adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:39845/0 (socket says 172.21.15.138:39845) 2024-03-29T11:21:41.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.863+0000 7fdb867fc640 1 -- 172.21.15.138:0/38643221 learned_addr learned my addr 172.21.15.138:0/38643221 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:41.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69804669 0 0) 0x7fdb880aaf60 con 0x7fdb8813ee90 2024-03-29T11:21:41.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:21:41.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb70003520 con 0x7fdb8813ee90 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981562172 0 0) 0x7fdb880afd10 con 0x7fdb880af900 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb880aaf60 con 0x7fdb880af900 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2836021433 0 0) 0x7fdb70003520 con 0x7fdb8813ee90 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 >> v1:172.21.15.152:6789/0 conn(0x7fdb880b4100 legacy=0x7fdb880a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 >> v1:172.21.15.138:6789/0 conn(0x7fdb880af900 legacy=0x7fdb8813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8813e970 con 0x7fdb8813ee90 2024-03-29T11:21:41.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.865+0000 7fdb85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb78001690 con 0x7fdb8813ee90 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 10 monclient: dump: 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:41.845 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb780018b0 con 0x7fdb8813ee90 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 1 -- 172.21.15.138:0/38643221 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb78001c40 con 0x7fdb8813ee90 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb85ffb640 10 monclient: dump: 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.846 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb8d650640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb8d650640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.866+0000 7fdb8d650640 4 set_mon_vals no callback set 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.867+0000 7fdb8d650640 10 monclient: shutdown 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.867+0000 7fdb8d650640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:41.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.867+0000 7fdb8d650640 1 -- 172.21.15.138:0/38643221 >> v1:172.21.15.138:6790/0 conn(0x7fdb8813ee90 legacy=0x7fdb880adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.871+0000 7fdb8d650640 1 -- 172.21.15.138:0/38643221 shutdown_connections 2024-03-29T11:21:41.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.871+0000 7fdb8d650640 1 -- 172.21.15.138:0/38643221 >> 172.21.15.138:0/38643221 conn(0x7fdb880b35e0 msgr2=0x7fdb880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:41.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.872+0000 7fdb8d650640 1 -- 172.21.15.138:0/38643221 shutdown_connections 2024-03-29T11:21:41.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.873+0000 7fdb8d650640 1 -- 172.21.15.138:0/38643221 wait complete. 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.874+0000 7fdb8d650640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.874+0000 7fdb8d650640 10 monclient: monmap: 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.849 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:41.850 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:41.850 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:41.850 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:41.850 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:41.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.874+0000 7fdb8d650640 1 librados: starting msgr at 2024-03-29T11:21:41.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.874+0000 7fdb8d650640 1 librados: starting objecter 2024-03-29T11:21:41.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.874+0000 7fdb8d650640 1 Processor -- start 2024-03-29T11:21:41.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 1 -- start start 2024-03-29T11:21:41.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 1 librados: setting wanted keys 2024-03-29T11:21:41.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 1 librados: calling monclient init 2024-03-29T11:21:41.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient: init 2024-03-29T11:21:41.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient(hunting): picked mon.noname-b con 0x7fdb880af900 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient(hunting): picked mon.noname-a con 0x7fdb880b4100 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient(hunting): picked mon.noname-c con 0x7fdb8813ee90 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8806e7a0 con 0x7fdb880b4100 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb88140a70 con 0x7fdb880af900 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb88140d50 con 0x7fdb8813ee90 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.875+0000 7fdb8d650640 10 monclient(hunting): authenticate will time out at 923.184003s 2024-03-29T11:21:41.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.878+0000 7fdb86ffd640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fdb880af900 0x7fdb88141a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:44247/0 (socket says 172.21.15.138:44247) 2024-03-29T11:21:41.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.878+0000 7fdb86ffd640 1 -- 172.21.15.138:0/4172879051 learned_addr learned my addr 172.21.15.138:0/4172879051 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:41.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.879+0000 7fdb6f7fe640 10 client.?.objecter ms_handle_connect 0x7fdb880af900 2024-03-29T11:21:41.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.879+0000 7fdb6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 client.?.objecter ms_handle_connect 0x7fdb8813ee90 2024-03-29T11:21:41.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536601226 0 0) 0x7fdb88140a70 con 0x7fdb880af900 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb74004eb0 con 0x7fdb880af900 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151115002 0 0) 0x7fdb88140d50 con 0x7fdb8813ee90 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.880+0000 7fdb6f7fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:21:41.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.881+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb88140a70 con 0x7fdb8813ee90 2024-03-29T11:21:41.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 client.?.objecter ms_handle_connect 0x7fdb880b4100 2024-03-29T11:21:41.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2978891177 0 0) 0x7fdb88140a70 con 0x7fdb8813ee90 2024-03-29T11:21:41.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb88140d50 con 0x7fdb8813ee90 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220022590 0 0) 0x7fdb8806e7a0 con 0x7fdb880b4100 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb88140a70 con 0x7fdb880b4100 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb80001b60 con 0x7fdb8813ee90 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:41.867 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189168884 0 0) 0x7fdb74004eb0 con 0x7fdb880af900 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb8806e7a0 con 0x7fdb880af900 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb7c004080 con 0x7fdb880af900 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.883+0000 7fdb6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3842506816 0 0) 0x7fdb88140d50 con 0x7fdb8813ee90 2024-03-29T11:21:41.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 >> v1:172.21.15.152:6789/0 conn(0x7fdb880af900 legacy=0x7fdb88141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 >> v1:172.21.15.138:6789/0 conn(0x7fdb880b4100 legacy=0x7fdb880719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 1 monclient: found mon.c 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88141030 con 0x7fdb8813ee90 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb80002010 con 0x7fdb8813ee90 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.884+0000 7fdb857fa640 4 set_mon_vals no callback set 2024-03-29T11:21:41.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb80003b30 con 0x7fdb8813ee90 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb6f7fe640 10 monclient: dump: 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:41.871 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb8d650640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb8d650640 10 monclient: _renew_subs 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb8d650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.885+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb88131c50 con 0x7fdb8813ee90 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.886+0000 7fdb8d650640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.886+0000 7fdb8d650640 10 monclient: _renew_subs 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.886+0000 7fdb8d650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.886+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb881318a0 con 0x7fdb8813ee90 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.886+0000 7fdb8d650640 1 librados: init done 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.888+0000 7fdb8d650640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.888+0000 7fdb8d650640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.888+0000 7fdb8d650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.888+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8806dfd0 con 0x7fdb8813ee90 2024-03-29T11:21:41.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.889+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fdb80019b00 con 0x7fdb8813ee90 2024-03-29T11:21:41.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.889+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fdb80083ac0 con 0x7fdb8813ee90 2024-03-29T11:21:41.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.889+0000 7fdb6f7fe640 10 client.4301.objecter ms_dispatch 0x7fdb88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:41.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.889+0000 7fdb6f7fe640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:41.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.889+0000 7fdb6f7fe640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:41.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.889+0000 7fdb6f7fe640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:21:41.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.890+0000 7fdb86ffd640 10 monclient: get_auth_request con 0x7fdb7406df50 auth_method 0 2024-03-29T11:21:41.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.891+0000 7fdb6f7fe640 10 client.4301.objecter ms_handle_connect 0x7fdb7406df50 2024-03-29T11:21:41.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.901+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb8004f3a0 con 0x7fdb8813ee90 2024-03-29T11:21:41.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.901+0000 7fdb6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:41.901+0000 7fdb6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:42.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.225+0000 7fdb8d650640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.225+0000 7fdb8d650640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.225+0000 7fdb8d650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:21:42.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.225+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb88141730 con 0x7fdb8813ee90 2024-03-29T11:21:42.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.228+0000 7fdb6f7fe640 1 -- 172.21.15.138:0/4172879051 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fdb800512d0 con 0x7fdb8813ee90 2024-03-29T11:21:42.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.228+0000 7fdb6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.228+0000 7fdb6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:42.204 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:42.204 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:42.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.232+0000 7fdb8d650640 10 librados: watch_flush enter 2024-03-29T11:21:42.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.232+0000 7fdb8d650640 10 librados: watch_flush exit 2024-03-29T11:21:42.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.232+0000 7fdb8d650640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:21:42.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.232+0000 7fdb8d650640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:21:42.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.234+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 >> v1:172.21.15.152:6816/3052585572 conn(0x7fdb7406df50 legacy=0x7fdb74070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.234+0000 7fdb8d650640 10 monclient: shutdown 2024-03-29T11:21:42.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.234+0000 7fdb8d650640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:42.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.234+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 >> v1:172.21.15.138:6790/0 conn(0x7fdb8813ee90 legacy=0x7fdb88072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.234+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 shutdown_connections 2024-03-29T11:21:42.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.234+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 >> 172.21.15.138:0/4172879051 conn(0x7fdb880b35e0 msgr2=0x7fdb880b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:42.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.234+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 shutdown_connections 2024-03-29T11:21:42.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.235+0000 7fdb8d650640 1 -- 172.21.15.138:0/4172879051 wait complete. 2024-03-29T11:21:42.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:42.235+0000 7fdb8d650640 1 librados: shutdown 2024-03-29T11:21:42.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:49.039 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.213+0000 7ffaf3f3d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.213+0000 7ffaf3f3d640 10 monclient: build_initial_monmap 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.213+0000 7ffaf3f3d640 10 monclient: monmap: 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:49.214442+0000 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:49.214442+0000 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:49.191 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:49.192 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:49.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.214+0000 7ffaf3f3d640 1 Processor -- start 2024-03-29T11:21:49.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.214+0000 7ffaf3f3d640 1 -- start start 2024-03-29T11:21:49.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.214+0000 7ffaf3f3d640 10 monclient: init 2024-03-29T11:21:49.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec0a9930 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec149dc0 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec14ff40 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0aae60 con 0x7ffaec14ff40 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0b4310 con 0x7ffaec0a9930 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0afdd0 con 0x7ffaec149dc0 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:49.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.215+0000 7ffaf3f3d640 10 monclient(hunting): authenticate will time out at 930.524613s 2024-03-29T11:21:49.196 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.219+0000 7ffaf24da640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ffaec14ff40 0x7ffaec152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38775/0 (socket says 172.21.15.138:38775) 2024-03-29T11:21:49.196 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.219+0000 7ffaf24da640 1 -- 172.21.15.138:0/281868631 learned_addr learned my addr 172.21.15.138:0/281868631 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:49.197 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985275202 0 0) 0x7ffaec0aae60 con 0x7ffaec14ff40 2024-03-29T11:21:49.197 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.197 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.197 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.197 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:49.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003520 con 0x7ffaec14ff40 2024-03-29T11:21:49.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2511684487 0 0) 0x7ffad8003520 con 0x7ffaec14ff40 2024-03-29T11:21:49.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 >> v1:172.21.15.138:6790/0 conn(0x7ffaec149dc0 legacy=0x7ffaec14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:49.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 >> v1:172.21.15.152:6789/0 conn(0x7ffaec0a9930 legacy=0x7ffaec143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: _finish_hunting 0 2024-03-29T11:21:49.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 monclient: found mon.noname-a 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec14a950 con 0x7ffaec14ff40 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: _finish_auth 0 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: _check_auth_tickets 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf3f3d640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf3f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffae4001a10 con 0x7ffaec14ff40 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: dump: 2024-03-29T11:21:49.199 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf3f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae4001ca0 con 0x7ffaec14ff40 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf0cd7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf3f3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.221+0000 7ffaf3f3d640 4 set_mon_vals no callback set 2024-03-29T11:21:49.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf0cd7640 1 -- 172.21.15.138:0/281868631 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffae4001a10 con 0x7ffaec14ff40 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf0cd7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf0cd7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf0cd7640 10 monclient: dump: 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:49.201 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:49.203 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf3f3d640 10 monclient: shutdown 2024-03-29T11:21:49.203 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf3f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:49.203 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/281868631 >> v1:172.21.15.138:6789/0 conn(0x7ffaec14ff40 legacy=0x7ffaec152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.203 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/281868631 shutdown_connections 2024-03-29T11:21:49.203 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.222+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/281868631 >> 172.21.15.138:0/281868631 conn(0x7ffaec0b35e0 msgr2=0x7ffaec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:49.203 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.223+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/281868631 shutdown_connections 2024-03-29T11:21:49.203 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.223+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/281868631 wait complete. 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.224+0000 7ffaf3f3d640 10 monclient: build_initial_monmap 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.224+0000 7ffaf3f3d640 10 monclient: monmap: 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.224+0000 7ffaf3f3d640 1 librados: starting msgr at 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.224+0000 7ffaf3f3d640 1 librados: starting objecter 2024-03-29T11:21:49.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.224+0000 7ffaf3f3d640 1 Processor -- start 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 1 -- start start 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 1 librados: setting wanted keys 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 1 librados: calling monclient init 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient: init 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec0a9930 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec14ff40 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec070270 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec071200 con 0x7ffaec14ff40 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec070d70 con 0x7ffaec070270 2024-03-29T11:21:49.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0706e0 con 0x7ffaec0a9930 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.225+0000 7ffaf3f3d640 10 monclient(hunting): authenticate will time out at 930.535009s 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.229+0000 7ffaf1cd9640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7ffaec0a9930 0x7ffaec1dea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:34305/0 (socket says 172.21.15.138:34305) 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.229+0000 7ffaf1cd9640 1 -- 172.21.15.138:0/2501718425 learned_addr learned my addr 172.21.15.138:0/2501718425 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.229+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec0a9930 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.229+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079237616 0 0) 0x7ffaec0706e0 con 0x7ffaec0a9930 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0004eb0 con 0x7ffaec0a9930 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec14ff40 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897875861 0 0) 0x7ffaec071200 con 0x7ffaec14ff40 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.230+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec0706e0 con 0x7ffaec14ff40 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.231+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1901772576 0 0) 0x7ffaec0706e0 con 0x7ffaec14ff40 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.231+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad0005490 con 0x7ffaec14ff40 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.231+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffadc001790 con 0x7ffaec14ff40 2024-03-29T11:21:49.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.231+0000 7ffae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.231+0000 7ffae27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.231+0000 7ffae27fc640 10 monclient(hunting): dump: 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.231+0000 7ffae27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:49.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1064184903 0 0) 0x7ffad0005490 con 0x7ffaec14ff40 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 >> v1:172.21.15.138:6790/0 conn(0x7ffaec0a9930 legacy=0x7ffaec1dea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 >> v1:172.21.15.152:6789/0 conn(0x7ffaec070270 legacy=0x7ffaec11eb30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 1 monclient: found mon.a 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec070870 con 0x7ffaec14ff40 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 10 monclient: _renew_subs 2024-03-29T11:21:49.214 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec0ac160 con 0x7ffaec14ff40 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 10 monclient: _renew_subs 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec1df190 con 0x7ffaec14ff40 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.232+0000 7ffaf3f3d640 1 librados: init done 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.234+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffadc001c80 con 0x7ffaec14ff40 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.234+0000 7ffae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.234+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffadc001790 con 0x7ffaec14ff40 2024-03-29T11:21:49.215 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.234+0000 7ffae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.234+0000 7ffae27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.234+0000 7ffae27fc640 10 monclient: dump: 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.218 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:49.219 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:49.219 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:49.219 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:49.219 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.234+0000 7ffae3fff640 4 set_mon_vals no callback set 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.235+0000 7ffaf3f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.235+0000 7ffaf3f3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.235+0000 7ffaf3f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.235+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad4005080 con 0x7ffaec14ff40 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.236+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ffadc0194e0 con 0x7ffaec14ff40 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.241+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ffadc083280 con 0x7ffaec14ff40 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.241+0000 7ffae27fc640 10 client.4305.objecter ms_dispatch 0x7ffaec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.241+0000 7ffae27fc640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.241+0000 7ffae27fc640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:49.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.241+0000 7ffae27fc640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-29T11:21:49.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.242+0000 7ffaf1cd9640 10 monclient: get_auth_request con 0x7ffad006e290 auth_method 0 2024-03-29T11:21:49.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.242+0000 7ffae27fc640 10 client.4305.objecter ms_handle_connect 0x7ffad006e290 2024-03-29T11:21:49.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.244+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffadc04eb60 con 0x7ffaec14ff40 2024-03-29T11:21:49.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.244+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.244+0000 7ffae27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:49.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.571+0000 7ffaf3f3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.571+0000 7ffaf3f3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.571+0000 7ffaf3f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:49.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.571+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad4005330 con 0x7ffaec14ff40 2024-03-29T11:21:49.551 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:49.551 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:49.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.572+0000 7ffae27fc640 1 -- 172.21.15.138:0/2501718425 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ffadc050a90 con 0x7ffaec14ff40 2024-03-29T11:21:49.552 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.572+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.552 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.572+0000 7ffae27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:49.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 10 librados: watch_flush enter 2024-03-29T11:21:49.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 10 librados: watch_flush exit 2024-03-29T11:21:49.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-29T11:21:49.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 10 client.4305.objecter successfully canceled tick 2024-03-29T11:21:49.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 >> v1:172.21.15.152:6816/3052585572 conn(0x7ffad006e290 legacy=0x7ffad0070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 10 monclient: shutdown 2024-03-29T11:21:49.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:49.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 >> v1:172.21.15.138:6789/0 conn(0x7ffaec14ff40 legacy=0x7ffaec11e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 shutdown_connections 2024-03-29T11:21:49.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.577+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 >> 172.21.15.138:0/2501718425 conn(0x7ffaec0b35e0 msgr2=0x7ffaec0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:49.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.581+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 shutdown_connections 2024-03-29T11:21:49.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.581+0000 7ffaf3f3d640 1 -- 172.21.15.138:0/2501718425 wait complete. 2024-03-29T11:21:49.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:49.581+0000 7ffaf3f3d640 1 librados: shutdown 2024-03-29T11:21:49.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:56.383 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:56.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.563+0000 7f747a43c640 10 monclient: get_monmap_and_config 2024-03-29T11:21:56.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.563+0000 7f747a43c640 10 monclient: build_initial_monmap 2024-03-29T11:21:56.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.563+0000 7f747a43c640 10 monclient: monmap: 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:21:56.565013+0000 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:21:56.565013+0000 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:21:56.541 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:56.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.565+0000 7f747a43c640 1 Processor -- start 2024-03-29T11:21:56.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.565+0000 7f747a43c640 1 -- start start 2024-03-29T11:21:56.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.565+0000 7f747a43c640 10 monclient: init 2024-03-29T11:21:56.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:56.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:56.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 10 monclient(hunting): picked mon.noname-c con 0x7f74740b0500 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:56.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 10 monclient(hunting): picked mon.noname-a con 0x7f74740b08e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:56.548 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 10 monclient(hunting): picked mon.noname-b con 0x7f74740aec70 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:56.549 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740ae7e0 con 0x7f74740b08e0 2024-03-29T11:21:56.549 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747413daf0 con 0x7f74740b0500 2024-03-29T11:21:56.549 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740accf0 con 0x7f74740aec70 2024-03-29T11:21:56.549 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:56.549 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.566+0000 7f747a43c640 10 monclient(hunting): authenticate will time out at 937.875746s 2024-03-29T11:21:56.549 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.571+0000 7f74737fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f74740b08e0 0x7f747413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:35711/0 (socket says 172.21.15.138:35711) 2024-03-29T11:21:56.549 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.572+0000 7f74737fe640 1 -- 172.21.15.138:0/3981521420 learned_addr learned my addr 172.21.15.138:0/3981521420 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:56.550 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.573+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29123565 0 0) 0x7f74740ae7e0 con 0x7f74740b08e0 2024-03-29T11:21:56.550 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.573+0000 7f7472ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.573+0000 7f7472ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.573+0000 7f7472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.573+0000 7f7472ffd640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:56.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.573+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c003520 con 0x7f74740b08e0 2024-03-29T11:21:56.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.574+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610723873 0 0) 0x7f745c003520 con 0x7f74740b08e0 2024-03-29T11:21:56.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.574+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 >> v1:172.21.15.138:6790/0 conn(0x7f74740aec70 legacy=0x7f747413ff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:56.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 >> v1:172.21.15.152:6789/0 conn(0x7f74740b0500 legacy=0x7f7474072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:56.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:56.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 1 monclient: found mon.noname-a 2024-03-29T11:21:56.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:56.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:21:56.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7474140620 con 0x7f74740b08e0 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7468001670 con 0x7f74740b08e0 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.575+0000 7f7472ffd640 10 monclient: dump: 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:56.554 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:56.555 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:56.555 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:56.555 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:56.555 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.576+0000 7f747a43c640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:21:56.555 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.576+0000 7f747a43c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:56.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74680018d0 con 0x7f74740b08e0 2024-03-29T11:21:56.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f7472ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:56.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f747a43c640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:56.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f747a43c640 4 set_mon_vals no callback set 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f7472ffd640 1 -- 172.21.15.138:0/3981521420 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7468001d20 con 0x7f74740b08e0 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f7472ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f7472ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.579+0000 7f7472ffd640 10 monclient: dump: 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:56.557 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:56.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.580+0000 7f747a43c640 10 monclient: shutdown 2024-03-29T11:21:56.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.580+0000 7f747a43c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.580+0000 7f747a43c640 1 -- 172.21.15.138:0/3981521420 >> v1:172.21.15.138:6789/0 conn(0x7f74740b08e0 legacy=0x7f747413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.584+0000 7f747a43c640 1 -- 172.21.15.138:0/3981521420 shutdown_connections 2024-03-29T11:21:56.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.584+0000 7f747a43c640 1 -- 172.21.15.138:0/3981521420 >> 172.21.15.138:0/3981521420 conn(0x7f747406b660 msgr2=0x7f747406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.585+0000 7f747a43c640 1 -- 172.21.15.138:0/3981521420 shutdown_connections 2024-03-29T11:21:56.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.585+0000 7f747a43c640 1 -- 172.21.15.138:0/3981521420 wait complete. 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.586+0000 7f747a43c640 10 monclient: build_initial_monmap 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.586+0000 7f747a43c640 10 monclient: monmap: 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:56.565 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.586+0000 7f747a43c640 1 librados: starting msgr at 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.586+0000 7f747a43c640 1 librados: starting objecter 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.587+0000 7f747a43c640 1 Processor -- start 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.587+0000 7f747a43c640 1 -- start start 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.587+0000 7f747a43c640 1 librados: setting wanted keys 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.587+0000 7f747a43c640 1 librados: calling monclient init 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.587+0000 7f747a43c640 10 monclient: init 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.588+0000 7f747a43c640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:56.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.588+0000 7f747a43c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:56.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.588+0000 7f747a43c640 10 monclient(hunting): picked mon.noname-b con 0x7f74740aec70 addr v1:172.21.15.152:6789/0 2024-03-29T11:21:56.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.591+0000 7f747a43c640 10 monclient(hunting): picked mon.noname-c con 0x7f74740b0500 addr v1:172.21.15.138:6790/0 2024-03-29T11:21:56.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.593+0000 7f747a43c640 10 monclient(hunting): picked mon.noname-a con 0x7f747412f730 addr v1:172.21.15.138:6789/0 2024-03-29T11:21:56.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.593+0000 7f747a43c640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7474130ab0 con 0x7f747412f730 2024-03-29T11:21:56.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.593+0000 7f747a43c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7474070af0 con 0x7f74740aec70 2024-03-29T11:21:56.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.593+0000 7f747a43c640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7474070520 con 0x7f74740b0500 2024-03-29T11:21:56.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.593+0000 7f747a43c640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:56.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.593+0000 7f747a43c640 10 monclient(hunting): authenticate will time out at 937.902156s 2024-03-29T11:21:56.573 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.592+0000 7f7473fff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f74740aec70 0x7f747412b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:41341/0 (socket says 172.21.15.138:41341) 2024-03-29T11:21:56.573 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.596+0000 7f7473fff640 1 -- 172.21.15.138:0/2402411963 learned_addr learned my addr 172.21.15.138:0/2402411963 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.597+0000 7f7470ff9640 10 client.?.objecter ms_handle_connect 0x7f74740aec70 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.597+0000 7f7470ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.597+0000 7f7470ff9640 10 client.?.objecter ms_handle_connect 0x7f747412f730 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.597+0000 7f7470ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.597+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151389929 0 0) 0x7f7474130ab0 con 0x7f747412f730 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.597+0000 7f7470ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.597+0000 7f7470ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460004eb0 con 0x7f747412f730 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 client.?.objecter ms_handle_connect 0x7f74740b0500 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3041239257 0 0) 0x7f7460004eb0 con 0x7f747412f730 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7460005300 con 0x7f747412f730 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f746c001a60 con 0x7f747412f730 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:56.581 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888856479 0 0) 0x7f7474070af0 con 0x7f74740aec70 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.598+0000 7f7470ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:21:56.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460005850 con 0x7f74740aec70 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156095450 0 0) 0x7f7474070520 con 0x7f74740b0500 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7474070af0 con 0x7f74740b0500 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3565372605 0 0) 0x7f7460005850 con 0x7f74740aec70 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7474070520 con 0x7f74740aec70 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7464003fc0 con 0x7f74740aec70 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:56.583 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.599+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4036619957 0 0) 0x7f7460005300 con 0x7f747412f730 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 >> v1:172.21.15.138:6790/0 conn(0x7f74740b0500 legacy=0x7f74741d5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 >> v1:172.21.15.152:6789/0 conn(0x7f74740aec70 legacy=0x7f747412b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:56.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 1 monclient: found mon.a 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74740706b0 con 0x7f747412f730 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f7470ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 10 monclient: _renew_subs 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747412bd90 con 0x7f747412f730 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 10 monclient: _renew_subs 2024-03-29T11:21:56.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.600+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f747412ece0 con 0x7f747412f730 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.601+0000 7f747a43c640 1 librados: init done 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.601+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f746c001fb0 con 0x7f747412f730 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.601+0000 7f7470ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.601+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f746c001a60 con 0x7f747412f730 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.601+0000 7f7470ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.601+0000 7f7470ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.601+0000 7f7470ff9640 10 monclient: dump: 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:21:56.586 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.602+0000 7f74727fc640 4 set_mon_vals no callback set 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.602+0000 7f747a43c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.603+0000 7f747a43c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.603+0000 7f747a43c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:56.587 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.603+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7454005080 con 0x7f747412f730 2024-03-29T11:21:56.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.606+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f746c0197e0 con 0x7f747412f730 2024-03-29T11:21:56.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.606+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f746c0837f0 con 0x7f747412f730 2024-03-29T11:21:56.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.606+0000 7f7470ff9640 10 client.4311.objecter ms_dispatch 0x7f7474000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:56.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.607+0000 7f7470ff9640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:56.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.607+0000 7f7470ff9640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:56.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.607+0000 7f7470ff9640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T11:21:56.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.607+0000 7f7473fff640 10 monclient: get_auth_request con 0x7f746006e4e0 auth_method 0 2024-03-29T11:21:56.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.611+0000 7f7470ff9640 10 client.4311.objecter ms_handle_connect 0x7f746006e4e0 2024-03-29T11:21:56.591 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.614+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f746c04ef00 con 0x7f747412f730 2024-03-29T11:21:56.591 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.614+0000 7f7470ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.591 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.614+0000 7f7470ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:56.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.939+0000 7f747a43c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.939+0000 7f747a43c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.939+0000 7f747a43c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:21:56.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.939+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7454005330 con 0x7f747412f730 2024-03-29T11:21:56.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.944+0000 7f7470ff9640 1 -- 172.21.15.138:0/2402411963 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f746c051000 con 0x7f747412f730 2024-03-29T11:21:56.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.944+0000 7f7470ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.944+0000 7f7470ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:56.922 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:21:56.922 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.948+0000 7f747a43c640 10 librados: watch_flush enter 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.948+0000 7f747a43c640 10 librados: watch_flush exit 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.948+0000 7f747a43c640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.948+0000 7f747a43c640 10 client.4311.objecter successfully canceled tick 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 >> v1:172.21.15.152:6816/3052585572 conn(0x7f746006e4e0 legacy=0x7f74600709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 10 monclient: shutdown 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 >> v1:172.21.15.138:6789/0 conn(0x7f747412f730 legacy=0x7f74741e0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 shutdown_connections 2024-03-29T11:21:56.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 >> 172.21.15.138:0/2402411963 conn(0x7f747406b660 msgr2=0x7f7474071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 shutdown_connections 2024-03-29T11:21:56.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 1 -- 172.21.15.138:0/2402411963 wait complete. 2024-03-29T11:21:56.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:21:56.949+0000 7f747a43c640 1 librados: shutdown 2024-03-29T11:21:56.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:03.751 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:03.908 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.931+0000 7f169a246640 10 monclient: get_monmap_and_config 2024-03-29T11:22:03.908 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.931+0000 7f169a246640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.908 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.931+0000 7f169a246640 10 monclient: monmap: 2024-03-29T11:22:03.908 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:03.908 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:03.933137+0000 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:03.933137+0000 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:03.909 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:03.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.936+0000 7f169a246640 1 Processor -- start 2024-03-29T11:22:03.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.937+0000 7f169a246640 1 -- start start 2024-03-29T11:22:03.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.937+0000 7f169a246640 10 monclient: init 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 10 monclient(hunting): picked mon.noname-b con 0x7f16940a9750 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 10 monclient(hunting): picked mon.noname-a con 0x7f16940add40 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 10 monclient(hunting): picked mon.noname-c con 0x7f16940aad20 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1694149870 con 0x7f16940add40 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16940abf80 con 0x7f16940aad20 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1694149ac0 con 0x7f16940a9750 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.938+0000 7f169a246640 10 monclient(hunting): authenticate will time out at 945.247299s 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.942+0000 7f1698a43640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f16940add40 0x7f1694143610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:57149/0 (socket says 172.21.15.138:57149) 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.942+0000 7f1698a43640 1 -- 172.21.15.138:0/3588596876 learned_addr learned my addr 172.21.15.138:0/3588596876 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.943+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155544148 0 0) 0x7f1694149870 con 0x7f16940add40 2024-03-29T11:22:03.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.943+0000 7f168bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.943+0000 7f168bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.943+0000 7f168bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.943+0000 7f168bfff640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:22:03.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.943+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1680003520 con 0x7f16940add40 2024-03-29T11:22:03.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 739593283 0 0) 0x7f1680003520 con 0x7f16940add40 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 >> v1:172.21.15.138:6790/0 conn(0x7f16940a9750 legacy=0x7f1694149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 >> v1:172.21.15.152:6789/0 conn(0x7f16940aad20 legacy=0x7f1694151bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 1 monclient: found mon.noname-a 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169414a6f0 con 0x7f16940add40 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f169a246640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f169a246640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f168c001a10 con 0x7f16940add40 2024-03-29T11:22:03.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f168bfff640 10 monclient: dump: 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:03.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.944+0000 7f169a246640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f168c001be0 con 0x7f16940add40 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f168bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f168bfff640 1 -- 172.21.15.138:0/3588596876 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f168c002030 con 0x7f16940add40 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f168bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f168bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f168bfff640 10 monclient: dump: 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:03.925 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:03.926 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:03.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f169a246640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:03.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.948+0000 7f169a246640 4 set_mon_vals no callback set 2024-03-29T11:22:03.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.949+0000 7f169a246640 10 monclient: shutdown 2024-03-29T11:22:03.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.949+0000 7f169a246640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:03.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.949+0000 7f169a246640 1 -- 172.21.15.138:0/3588596876 >> v1:172.21.15.138:6789/0 conn(0x7f16940add40 legacy=0x7f1694143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.950+0000 7f169a246640 1 -- 172.21.15.138:0/3588596876 shutdown_connections 2024-03-29T11:22:03.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.950+0000 7f169a246640 1 -- 172.21.15.138:0/3588596876 >> 172.21.15.138:0/3588596876 conn(0x7f16940673e0 msgr2=0x7f16940b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:03.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.955+0000 7f169a246640 1 -- 172.21.15.138:0/3588596876 shutdown_connections 2024-03-29T11:22:03.932 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.955+0000 7f169a246640 1 -- 172.21.15.138:0/3588596876 wait complete. 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.956+0000 7f169a246640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.956+0000 7f169a246640 10 monclient: monmap: 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:03.933 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.957+0000 7f169a246640 1 librados: starting msgr at 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.957+0000 7f169a246640 1 librados: starting objecter 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.957+0000 7f169a246640 1 Processor -- start 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.957+0000 7f169a246640 1 -- start start 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.957+0000 7f169a246640 1 librados: setting wanted keys 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.957+0000 7f169a246640 1 librados: calling monclient init 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.957+0000 7f169a246640 10 monclient: init 2024-03-29T11:22:03.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.958+0000 7f169a246640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.959+0000 7f169a246640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:03.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.959+0000 7f169a246640 10 monclient(hunting): picked mon.noname-b con 0x7f16940a9750 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:03.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.962+0000 7f169a246640 10 monclient(hunting): picked mon.noname-c con 0x7f16940aad20 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.963+0000 7f1699244640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f16940a9750 0x7f16941dec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:34401/0 (socket says 172.21.15.138:34401) 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.963+0000 7f1699244640 1 -- 172.21.15.138:0/824611966 learned_addr learned my addr 172.21.15.138:0/824611966 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.964+0000 7f169a246640 10 monclient(hunting): picked mon.noname-a con 0x7f16940add40 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.964+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16941de7e0 con 0x7f16940add40 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.964+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169406db10 con 0x7f16940a9750 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16941df380 con 0x7f16940aad20 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f169a246640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f169a246640 10 monclient(hunting): authenticate will time out at 945.274065s 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 10 client.?.objecter ms_handle_connect 0x7f16940a9750 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887382670 0 0) 0x7f169406db10 con 0x7f16940a9750 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678004eb0 con 0x7f16940a9750 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4103168688 0 0) 0x7f1678004eb0 con 0x7f16940a9750 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f169406db10 con 0x7f16940a9750 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.965+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1684001960 con 0x7f16940a9750 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.945 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2314919103 0 0) 0x7f169406db10 con 0x7f16940a9750 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 >> v1:172.21.15.138:6790/0 conn(0x7f16940aad20 legacy=0x7f169413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 >> v1:172.21.15.138:6789/0 conn(0x7f16940add40 legacy=0x7f1694147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 1 monclient: found mon.b 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16941df510 con 0x7f16940a9750 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f1689ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f169a246640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f169a246640 10 monclient: _renew_subs 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f169a246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1694144170 con 0x7f16940a9750 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.966+0000 7f169a246640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f169a246640 10 monclient: _renew_subs 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f169a246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f169413ca40 con 0x7f16940a9750 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f169a246640 1 librados: init done 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1684001d40 con 0x7f16940a9750 2024-03-29T11:22:03.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f1689ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1684001bc0 con 0x7f16940a9750 2024-03-29T11:22:03.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f168affd640 4 set_mon_vals no callback set 2024-03-29T11:22:03.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.967+0000 7f1689ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.969+0000 7f1689ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.969+0000 7f169a246640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.969+0000 7f1689ffb640 10 monclient: dump: 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:03.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.969+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f1684003b40 con 0x7f16940a9750 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f169a246640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f169a246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1674005080 con 0x7f16940a9750 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f1698a43640 10 monclient: get_auth_request con 0x7f167806dd20 auth_method 0 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f1684084b30 con 0x7f16940a9750 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f1689ffb640 10 client.4291.objecter ms_dispatch 0x7f1694000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f1689ffb640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f1689ffb640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f1689ffb640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T11:22:03.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.970+0000 7f1689ffb640 10 client.4291.objecter ms_handle_connect 0x7f167806dd20 2024-03-29T11:22:03.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.978+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1684050410 con 0x7f16940a9750 2024-03-29T11:22:03.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.978+0000 7f1689ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:03.979+0000 7f1689ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:04.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.314+0000 7f169a246640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.314+0000 7f169a246640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.314+0000 7f169a246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:22:04.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.314+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1674005330 con 0x7f16940a9750 2024-03-29T11:22:04.293 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.316+0000 7f1689ffb640 1 -- 172.21.15.138:0/824611966 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f168401e8f0 con 0x7f16940a9750 2024-03-29T11:22:04.293 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.316+0000 7f1689ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.293 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.316+0000 7f1689ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:04.295 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:04.295 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:04.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.320+0000 7f169a246640 10 librados: watch_flush enter 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.321+0000 7f169a246640 10 librados: watch_flush exit 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.321+0000 7f169a246640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.321+0000 7f169a246640 10 client.4291.objecter successfully canceled tick 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 >> v1:172.21.15.152:6816/3052585572 conn(0x7f167806dd20 legacy=0x7f16780701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 10 monclient: shutdown 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 >> v1:172.21.15.152:6789/0 conn(0x7f16940a9750 legacy=0x7f16941dec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 shutdown_connections 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 >> 172.21.15.138:0/824611966 conn(0x7f16940673e0 msgr2=0x7f1694069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 shutdown_connections 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 1 -- 172.21.15.138:0/824611966 wait complete. 2024-03-29T11:22:04.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:04.322+0000 7f169a246640 1 librados: shutdown 2024-03-29T11:22:04.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:11.124 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:11.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.289+0000 7f26e5996640 10 monclient: get_monmap_and_config 2024-03-29T11:22:11.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.289+0000 7f26e5996640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.289+0000 7f26e5996640 10 monclient: monmap: 2024-03-29T11:22:11.266 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:11.266 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:11.266 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:11.290637+0000 2024-03-29T11:22:11.267 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:11.290637+0000 2024-03-29T11:22:11.267 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.267 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:11.267 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:11.267 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:11.267 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:11.267 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:11.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.290+0000 7f26e5996640 1 Processor -- start 2024-03-29T11:22:11.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.290+0000 7f26e5996640 1 -- start start 2024-03-29T11:22:11.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.290+0000 7f26e5996640 10 monclient: init 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 10 monclient(hunting): picked mon.noname-a con 0x7f26e01457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 10 monclient(hunting): picked mon.noname-c con 0x7f26e013f650 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 10 monclient(hunting): picked mon.noname-b con 0x7f26e00a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00aaef0 con 0x7f26e01457f0 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e0145bd0 con 0x7f26e013f650 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e0145eb0 con 0x7f26e00a9d70 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26e5996640 10 monclient(hunting): authenticate will time out at 952.600390s 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26de7fc640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f26e013f650 0x7f26e00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:56564/0 (socket says 172.21.15.138:56564) 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26de7fc640 1 -- 172.21.15.138:0/1043353507 learned_addr learned my addr 172.21.15.138:0/1043353507 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.291+0000 7f26df7fe640 1 --1- 172.21.15.138:0/1043353507 >> v1:172.21.15.138:6790/0 conn(0x7f26e00a9d70 0x7f26e014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:48764/0 (socket says 172.21.15.138:48764) 2024-03-29T11:22:11.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135240478 0 0) 0x7f26e0145eb0 con 0x7f26e00a9d70 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8003520 con 0x7f26e00a9d70 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664835955 0 0) 0x7f26e00aaef0 con 0x7f26e01457f0 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e0145eb0 con 0x7f26e01457f0 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688234699 0 0) 0x7f26c8003520 con 0x7f26e00a9d70 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 >> v1:172.21.15.152:6789/0 conn(0x7f26e013f650 legacy=0x7f26e00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 >> v1:172.21.15.138:6789/0 conn(0x7f26e01457f0 legacy=0x7f26e013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0146340 con 0x7f26e00a9d70 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f26d8001690 con 0x7f26e00a9d70 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: dump: 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:11.273 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26e5996640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26e5996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26d8001870 con 0x7f26e00a9d70 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 1 -- 172.21.15.138:0/1043353507 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f26d8001c80 con 0x7f26e00a9d70 2024-03-29T11:22:11.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26ddffb640 10 monclient: dump: 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26e5996640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.292+0000 7f26e5996640 4 set_mon_vals no callback set 2024-03-29T11:22:11.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.293+0000 7f26e5996640 10 monclient: shutdown 2024-03-29T11:22:11.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.293+0000 7f26e5996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:11.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.293+0000 7f26e5996640 1 -- 172.21.15.138:0/1043353507 >> v1:172.21.15.138:6790/0 conn(0x7f26e00a9d70 legacy=0x7f26e014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.293+0000 7f26e5996640 1 -- 172.21.15.138:0/1043353507 shutdown_connections 2024-03-29T11:22:11.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.293+0000 7f26e5996640 1 -- 172.21.15.138:0/1043353507 >> 172.21.15.138:0/1043353507 conn(0x7f26e0066a00 msgr2=0x7f26e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:11.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.294+0000 7f26e5996640 1 -- 172.21.15.138:0/1043353507 shutdown_connections 2024-03-29T11:22:11.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.294+0000 7f26e5996640 1 -- 172.21.15.138:0/1043353507 wait complete. 2024-03-29T11:22:11.282 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.294+0000 7f26e5996640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.282 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.294+0000 7f26e5996640 10 monclient: monmap: 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.295+0000 7f26e5996640 1 librados: starting msgr at 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.295+0000 7f26e5996640 1 librados: starting objecter 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.295+0000 7f26e5996640 1 Processor -- start 2024-03-29T11:22:11.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.295+0000 7f26e5996640 1 -- start start 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.295+0000 7f26e5996640 1 librados: setting wanted keys 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.295+0000 7f26e5996640 1 librados: calling monclient init 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.295+0000 7f26e5996640 10 monclient: init 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 10 monclient(hunting): picked mon.noname-c con 0x7f26e00a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 10 monclient(hunting): picked mon.noname-b con 0x7f26e013f650 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 10 monclient(hunting): picked mon.noname-a con 0x7f26e01457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e006ef80 con 0x7f26e01457f0 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e006ed20 con 0x7f26e013f650 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e006fe00 con 0x7f26e00a9d70 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26e5996640 10 monclient(hunting): authenticate will time out at 952.605475s 2024-03-29T11:22:11.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26df7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f26e01457f0 0x7f26e0070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48202/0 (socket says 172.21.15.138:48202) 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26df7fe640 1 -- 172.21.15.138:0/3335550834 learned_addr learned my addr 172.21.15.138:0/3335550834 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 client.?.objecter ms_handle_connect 0x7f26e00a9d70 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 client.?.objecter ms_handle_connect 0x7f26e01457f0 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700888265 0 0) 0x7f26e006ef80 con 0x7f26e01457f0 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.296+0000 7f26c7fff640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26cc004eb0 con 0x7f26e01457f0 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852072469 0 0) 0x7f26e006fe00 con 0x7f26e00a9d70 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.285 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e006ef80 con 0x7f26e00a9d70 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2145914603 0 0) 0x7f26cc004eb0 con 0x7f26e01457f0 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26e006fe00 con 0x7f26e01457f0 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f26d8001bc0 con 0x7f26e01457f0 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): dump: 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.286 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.297+0000 7f26c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787714869 0 0) 0x7f26e006ef80 con 0x7f26e00a9d70 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26cc004eb0 con 0x7f26e00a9d70 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f26d4001890 con 0x7f26e00a9d70 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 monclient(hunting): dump: 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:11.287 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 client.?.objecter ms_handle_connect 0x7f26e013f650 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1535951329 0 0) 0x7f26cc004eb0 con 0x7f26e00a9d70 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 >> v1:172.21.15.152:6789/0 conn(0x7f26e013f650 legacy=0x7f26e0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 >> v1:172.21.15.138:6789/0 conn(0x7f26e01457f0 legacy=0x7f26e0070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 monclient: found mon.c 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00700e0 con 0x7f26e00a9d70 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 10 monclient: _renew_subs 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e006e800 con 0x7f26e00a9d70 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 10 monclient: _renew_subs 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:11.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e0142c30 con 0x7f26e00a9d70 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.298+0000 7f26e5996640 1 librados: init done 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.299+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26d40035f0 con 0x7f26e00a9d70 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.299+0000 7f26c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.299+0000 7f26dcff9640 4 set_mon_vals no callback set 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.299+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f26d4001d80 con 0x7f26e00a9d70 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.299+0000 7f26c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.299+0000 7f26c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.299+0000 7f26c7fff640 10 monclient: dump: 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.290 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.301+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f26d40197f0 con 0x7f26e00a9d70 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26de7fc640 10 monclient: get_auth_request con 0x7f26cc06df50 auth_method 0 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f26d40838c0 con 0x7f26e00a9d70 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26c7fff640 10 client.4313.objecter ms_dispatch 0x7f26e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26c7fff640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26c7fff640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26c7fff640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26c7fff640 10 client.4313.objecter ms_handle_connect 0x7f26cc06df50 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26e5996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26e5996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26e5996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:11.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.302+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c0005080 con 0x7f26e00a9d70 2024-03-29T11:22:11.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.310+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26d404efd0 con 0x7f26e00a9d70 2024-03-29T11:22:11.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.310+0000 7f26c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.310+0000 7f26c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:11.677 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.700+0000 7f26e5996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.678 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.700+0000 7f26e5996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.678 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.700+0000 7f26e5996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:11.678 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.700+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26c0005330 con 0x7f26e00a9d70 2024-03-29T11:22:11.678 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.701+0000 7f26c7fff640 1 -- 172.21.15.138:0/3335550834 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f26d40510d0 con 0x7f26e00a9d70 2024-03-29T11:22:11.678 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.701+0000 7f26c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.678 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.701+0000 7f26c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:11.678 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:11.679 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 10 librados: watch_flush enter 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 10 librados: watch_flush exit 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 10 client.4313.objecter successfully canceled tick 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 >> v1:172.21.15.152:6816/3052585572 conn(0x7f26cc06df50 legacy=0x7f26cc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 10 monclient: shutdown 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:11.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.704+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 >> v1:172.21.15.138:6790/0 conn(0x7f26e00a9d70 legacy=0x7f26e01402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.682 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.705+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 shutdown_connections 2024-03-29T11:22:11.682 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.705+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 >> 172.21.15.138:0/3335550834 conn(0x7f26e0066a00 msgr2=0x7f26e013ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:11.683 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.705+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 shutdown_connections 2024-03-29T11:22:11.683 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.705+0000 7f26e5996640 1 -- 172.21.15.138:0/3335550834 wait complete. 2024-03-29T11:22:11.683 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:11.705+0000 7f26e5996640 1 librados: shutdown 2024-03-29T11:22:11.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:18.503 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:18.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.659+0000 7ffbbf1b6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:18.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.659+0000 7ffbbf1b6640 10 monclient: build_initial_monmap 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.659+0000 7ffbbf1b6640 10 monclient: monmap: 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:18.661023+0000 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:18.661023+0000 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:18.637 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.660+0000 7ffbbf1b6640 1 Processor -- start 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.660+0000 7ffbbf1b6640 1 -- start start 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.660+0000 7ffbbf1b6640 10 monclient: init 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb81422b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb8142690 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb80b17d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb8142c80 con 0x7ffbb8142690 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb80aae40 con 0x7ffbb81422b0 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.661+0000 7ffbbf1b6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb813fbd0 con 0x7ffbb80b17d0 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbbf1b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:18.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbbf1b6640 10 monclient(hunting): authenticate will time out at 959.971071s 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb7fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ffbb8142690 0x7ffbb80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45886/0 (socket says 172.21.15.138:45886) 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb7fff640 1 -- 172.21.15.138:0/2327714611 learned_addr learned my addr 172.21.15.138:0/2327714611 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931119761 0 0) 0x7ffbb80aae40 con 0x7ffbb81422b0 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb77fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:22:18.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.662+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba0003520 con 0x7ffbb81422b0 2024-03-29T11:22:18.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489046884 0 0) 0x7ffbb813fbd0 con 0x7ffbb80b17d0 2024-03-29T11:22:18.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb80aae40 con 0x7ffbb80b17d0 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864278796 0 0) 0x7ffbb8142c80 con 0x7ffbb8142690 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb813fbd0 con 0x7ffbb8142690 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804639602 0 0) 0x7ffba0003520 con 0x7ffbb81422b0 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 >> v1:172.21.15.138:6790/0 conn(0x7ffbb80b17d0 legacy=0x7ffbb80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 >> v1:172.21.15.138:6789/0 conn(0x7ffbb8142690 legacy=0x7ffbb80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 monclient: found mon.noname-c 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.663+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8151940 con 0x7ffbb81422b0 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffbac001690 con 0x7ffbb81422b0 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: dump: 2024-03-29T11:22:18.644 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbbf1b6640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbbf1b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbac0017f0 con 0x7ffbb81422b0 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 1 -- 172.21.15.138:0/2327714611 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffbac001b80 con 0x7ffbb81422b0 2024-03-29T11:22:18.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbb77fe640 10 monclient: dump: 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbbf1b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.664+0000 7ffbbf1b6640 4 set_mon_vals no callback set 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.665+0000 7ffbbf1b6640 10 monclient: shutdown 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.665+0000 7ffbbf1b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.665+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/2327714611 >> v1:172.21.15.152:6789/0 conn(0x7ffbb81422b0 legacy=0x7ffbb813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.665+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/2327714611 shutdown_connections 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.665+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/2327714611 >> 172.21.15.138:0/2327714611 conn(0x7ffbb8066a00 msgr2=0x7ffbb8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.665+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/2327714611 shutdown_connections 2024-03-29T11:22:18.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.665+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/2327714611 wait complete. 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.666+0000 7ffbbf1b6640 10 monclient: build_initial_monmap 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.666+0000 7ffbbf1b6640 10 monclient: monmap: 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.667+0000 7ffbbf1b6640 1 librados: starting msgr at 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.667+0000 7ffbbf1b6640 1 librados: starting objecter 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.667+0000 7ffbbf1b6640 1 Processor -- start 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.667+0000 7ffbbf1b6640 1 -- start start 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.667+0000 7ffbbf1b6640 1 librados: setting wanted keys 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.667+0000 7ffbbf1b6640 1 librados: calling monclient init 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.667+0000 7ffbbf1b6640 10 monclient: init 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.668+0000 7ffbbf1b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.668+0000 7ffbbf1b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.668+0000 7ffbbf1b6640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb80b17d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbbf1b6640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb81422b0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbbf1b6640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb8142690 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbbf1b6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb80712c0 con 0x7ffbb81422b0 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbbf1b6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb806ed90 con 0x7ffbb8142690 2024-03-29T11:22:18.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbbf1b6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb80b13e0 con 0x7ffbb80b17d0 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbbf1b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbbf1b6640 10 monclient(hunting): authenticate will time out at 959.978303s 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb7fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ffbb81422b0 0x7ffbb806f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:46150/0 (socket says 172.21.15.138:46150) 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb7fff640 1 -- 172.21.15.138:0/95062504 learned_addr learned my addr 172.21.15.138:0/95062504 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 client.?.objecter ms_handle_connect 0x7ffbb81422b0 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 client.?.objecter ms_handle_connect 0x7ffbb8142690 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379144595 0 0) 0x7ffbb80712c0 con 0x7ffbb81422b0 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba4004eb0 con 0x7ffbb81422b0 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 client.?.objecter ms_handle_connect 0x7ffbb80b17d0 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.669+0000 7ffbb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216922178 0 0) 0x7ffbb80b13e0 con 0x7ffbb80b17d0 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb80712c0 con 0x7ffbb80b17d0 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020968079 0 0) 0x7ffbb806ed90 con 0x7ffbb8142690 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb80b13e0 con 0x7ffbb8142690 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279537182 0 0) 0x7ffba4004eb0 con 0x7ffbb81422b0 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbb806ed90 con 0x7ffbb81422b0 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffba8001870 con 0x7ffbb81422b0 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): dump: 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.660 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.670+0000 7ffbb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 672500689 0 0) 0x7ffbb80b13e0 con 0x7ffbb8142690 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffba4004eb0 con 0x7ffbb8142690 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffbb0001c30 con 0x7ffbb8142690 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient(hunting): dump: 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:18.661 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4292837086 0 0) 0x7ffbb80712c0 con 0x7ffbb80b17d0 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbb80b13e0 con 0x7ffbb80b17d0 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffbac001960 con 0x7ffbb80b17d0 2024-03-29T11:22:18.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient(hunting): dump: 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3414490175 0 0) 0x7ffbb806ed90 con 0x7ffbb81422b0 2024-03-29T11:22:18.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 >> v1:172.21.15.138:6790/0 conn(0x7ffbb80b17d0 legacy=0x7ffbb8071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 >> v1:172.21.15.152:6789/0 conn(0x7ffbb8142690 legacy=0x7ffbb813bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 monclient: found mon.a 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb80b10e0 con 0x7ffbb81422b0 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbbf1b6640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbbf1b6640 10 monclient: _renew_subs 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbbf1b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb806fea0 con 0x7ffbb81422b0 2024-03-29T11:22:18.664 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbbf1b6640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.671+0000 7ffbbf1b6640 10 monclient: _renew_subs 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.672+0000 7ffbbf1b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.672+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb806f490 con 0x7ffbb81422b0 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.672+0000 7ffbbf1b6640 1 librados: init done 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.672+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba80019d0 con 0x7ffbb81422b0 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.672+0000 7ffbb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.672+0000 7ffbb67fc640 4 set_mon_vals no callback set 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.673+0000 7ffbbf1b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.673+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffba8004980 con 0x7ffbb81422b0 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.673+0000 7ffbb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.673+0000 7ffbb57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.673+0000 7ffbb57fa640 10 monclient: dump: 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:18.665 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.674+0000 7ffbbf1b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.674+0000 7ffbbf1b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.674+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb98005080 con 0x7ffbb81422b0 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.675+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ffba8004b80 con 0x7ffbb81422b0 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.677+0000 7ffbbcf52640 10 monclient: get_auth_request con 0x7ffba406df60 auth_method 0 2024-03-29T11:22:18.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.677+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ffba80849b0 con 0x7ffbb81422b0 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.677+0000 7ffbb57fa640 10 client.4326.objecter ms_dispatch 0x7ffbb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.677+0000 7ffbb57fa640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.677+0000 7ffbb57fa640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.677+0000 7ffbb57fa640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.677+0000 7ffbb57fa640 10 client.4326.objecter ms_handle_connect 0x7ffba406df60 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.683+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffba80502f0 con 0x7ffbb81422b0 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.683+0000 7ffbb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.667 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:18.683+0000 7ffbb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:19.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.060+0000 7ffbbf1b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.060+0000 7ffbbf1b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.060+0000 7ffbbf1b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:19.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.060+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb98005330 con 0x7ffbb81422b0 2024-03-29T11:22:19.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.061+0000 7ffbb57fa640 1 -- 172.21.15.138:0/95062504 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ffba8052220 con 0x7ffbb81422b0 2024-03-29T11:22:19.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.061+0000 7ffbb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.061+0000 7ffbb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:19.038 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:19.038 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 10 librados: watch_flush enter 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 10 librados: watch_flush exit 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 10 client.4326.objecter successfully canceled tick 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 >> v1:172.21.15.152:6816/3052585572 conn(0x7ffba406df60 legacy=0x7ffba4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 10 monclient: shutdown 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:19.042 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 >> v1:172.21.15.138:6789/0 conn(0x7ffbb81422b0 legacy=0x7ffbb806f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 shutdown_connections 2024-03-29T11:22:19.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 >> 172.21.15.138:0/95062504 conn(0x7ffbb8066a00 msgr2=0x7ffbb80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:19.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 shutdown_connections 2024-03-29T11:22:19.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 1 -- 172.21.15.138:0/95062504 wait complete. 2024-03-29T11:22:19.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:19.065+0000 7ffbbf1b6640 1 librados: shutdown 2024-03-29T11:22:19.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:25.863 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.019+0000 7f80d28e2640 10 monclient: get_monmap_and_config 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.019+0000 7f80d28e2640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.019+0000 7f80d28e2640 10 monclient: monmap: 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:26.019593+0000 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:26.019593+0000 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:25.995 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:25.996 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:25.996 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:25.996 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.020+0000 7f80d28e2640 1 Processor -- start 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.020+0000 7f80d28e2640 1 -- start start 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.020+0000 7f80d28e2640 10 monclient: init 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.020+0000 7f80d28e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 10 monclient(hunting): picked mon.noname-b con 0x7f80cc1457f0 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 10 monclient(hunting): picked mon.noname-c con 0x7f80cc13f650 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80cbfff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f80cc1457f0 0x7f80cc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:45224/0 (socket says 172.21.15.138:45224) 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 10 monclient(hunting): picked mon.noname-a con 0x7f80cc0a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:25.999 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80cbfff640 1 -- 172.21.15.138:0/2632606450 learned_addr learned my addr 172.21.15.138:0/2632606450 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc0aaef0 con 0x7f80cc0a9d70 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc145bd0 con 0x7f80cc13f650 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc145eb0 con 0x7f80cc1457f0 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.021+0000 7f80d28e2640 10 monclient(hunting): authenticate will time out at 967.329733s 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754947119 0 0) 0x7f80cc145eb0 con 0x7f80cc1457f0 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b4003520 con 0x7f80cc1457f0 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369337599 0 0) 0x7f80cc145bd0 con 0x7f80cc13f650 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.001 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.001 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:22:26.001 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80cc145eb0 con 0x7f80cc13f650 2024-03-29T11:22:26.001 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1394898212 0 0) 0x7f80cc145eb0 con 0x7f80cc13f650 2024-03-29T11:22:26.001 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 >> v1:172.21.15.138:6790/0 conn(0x7f80cc1457f0 legacy=0x7f80cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.001 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.022+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 >> v1:172.21.15.138:6789/0 conn(0x7f80cc0a9d70 legacy=0x7f80cc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:26.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:26.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 1 monclient: found mon.noname-c 2024-03-29T11:22:26.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:26.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc146340 con 0x7f80cc13f650 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f80c0001670 con 0x7f80cc13f650 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: dump: 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.006 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80d28e2640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80d28e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c0001850 con 0x7f80cc13f650 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 1 -- 172.21.15.138:0/2632606450 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f80c0001ca0 con 0x7f80cc13f650 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80caffd640 10 monclient: dump: 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:26.007 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80d28e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.025+0000 7f80d28e2640 4 set_mon_vals no callback set 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.027+0000 7f80d28e2640 10 monclient: shutdown 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.027+0000 7f80d28e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.027+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 >> v1:172.21.15.152:6789/0 conn(0x7f80cc13f650 legacy=0x7f80cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.027+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 shutdown_connections 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.027+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 >> 172.21.15.138:0/2632606450 conn(0x7f80cc066a00 msgr2=0x7f80cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:26.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.027+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 shutdown_connections 2024-03-29T11:22:26.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.027+0000 7f80d28e2640 1 -- 172.21.15.138:0/2632606450 wait complete. 2024-03-29T11:22:26.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.028+0000 7f80d28e2640 10 monclient: build_initial_monmap 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.028+0000 7f80d28e2640 10 monclient: monmap: 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.028+0000 7f80d28e2640 1 librados: starting msgr at 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.028+0000 7f80d28e2640 1 librados: starting objecter 2024-03-29T11:22:26.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.029+0000 7f80d28e2640 1 Processor -- start 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.029+0000 7f80d28e2640 1 -- start start 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.029+0000 7f80d28e2640 1 librados: setting wanted keys 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.029+0000 7f80d28e2640 1 librados: calling monclient init 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.029+0000 7f80d28e2640 10 monclient: init 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 10 monclient(hunting): picked mon.noname-a con 0x7f80cc0a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 10 monclient(hunting): picked mon.noname-b con 0x7f80cc13f650 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 10 monclient(hunting): picked mon.noname-c con 0x7f80cc1457f0 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc142840 con 0x7f80cc0a9d70 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc142cd0 con 0x7f80cc13f650 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc0ac180 con 0x7f80cc1457f0 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:26.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d0e7f640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f80cc1457f0 0x7f80cc1e29e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:46792/0 (socket says 172.21.15.138:46792) 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d0e7f640 1 -- 172.21.15.138:0/2859032393 learned_addr learned my addr 172.21.15.138:0/2859032393 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 10 client.?.objecter ms_handle_connect 0x7f80cc1457f0 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80d28e2640 10 monclient(hunting): authenticate will time out at 967.339494s 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958078687 0 0) 0x7f80cc0ac180 con 0x7f80cc1457f0 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:22:26.018 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.031+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b8004eb0 con 0x7f80cc1457f0 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786479713 0 0) 0x7f80b8004eb0 con 0x7f80cc1457f0 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80cc0ac180 con 0x7f80cc1457f0 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f80c4001c60 con 0x7f80cc1457f0 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:26.019 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4231220412 0 0) 0x7f80cc0ac180 con 0x7f80cc1457f0 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 >> v1:172.21.15.152:6789/0 conn(0x7f80cc13f650 legacy=0x7f80cc0b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.032+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 >> v1:172.21.15.138:6789/0 conn(0x7f80cc0a9d70 legacy=0x7f80cc1d7180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 1 monclient: found mon.c 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc1417f0 con 0x7f80cc1457f0 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:26.020 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 10 monclient: _renew_subs 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc141f60 con 0x7f80cc1457f0 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 10 monclient: _renew_subs 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc142500 con 0x7f80cc1457f0 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80d28e2640 1 librados: init done 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c4001ee0 con 0x7f80cc1457f0 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f80c4003dc0 con 0x7f80cc1457f0 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80c8ff9640 10 monclient: dump: 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:26.022 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.033+0000 7f80ca7fc640 4 set_mon_vals no callback set 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.035+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f80c4003f20 con 0x7f80cc1457f0 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.035+0000 7f80d28e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.035+0000 7f80d28e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.035+0000 7f80d28e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.035+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ac005080 con 0x7f80cc1457f0 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.037+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f80c4084e80 con 0x7f80cc1457f0 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.037+0000 7f80c8ff9640 10 client.4325.objecter ms_dispatch 0x7f80cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.037+0000 7f80c8ff9640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.037+0000 7f80c8ff9640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.037+0000 7f80c8ff9640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.037+0000 7f80cbfff640 10 monclient: get_auth_request con 0x7f80b806dbc0 auth_method 0 2024-03-29T11:22:26.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.037+0000 7f80c8ff9640 10 client.4325.objecter ms_handle_connect 0x7f80b806dbc0 2024-03-29T11:22:26.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.043+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80c4050760 con 0x7f80cc1457f0 2024-03-29T11:22:26.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.043+0000 7f80c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.043+0000 7f80c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:26.322 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.346+0000 7f80d28e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.322 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.346+0000 7f80d28e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.346+0000 7f80d28e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:26.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.346+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80ac005330 con 0x7f80cc1457f0 2024-03-29T11:22:26.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.347+0000 7f80c8ff9640 1 -- 172.21.15.138:0/2859032393 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f80c4052690 con 0x7f80cc1457f0 2024-03-29T11:22:26.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.347+0000 7f80c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.347+0000 7f80c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:26.324 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:26.324 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.350+0000 7f80d28e2640 10 librados: watch_flush enter 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.350+0000 7f80d28e2640 10 librados: watch_flush exit 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.350+0000 7f80d28e2640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.350+0000 7f80d28e2640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.350+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 >> v1:172.21.15.152:6816/3052585572 conn(0x7f80b806dbc0 legacy=0x7f80b8070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 10 monclient: shutdown 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 >> v1:172.21.15.138:6790/0 conn(0x7f80cc1457f0 legacy=0x7f80cc1e29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 shutdown_connections 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 >> 172.21.15.138:0/2859032393 conn(0x7f80cc066a00 msgr2=0x7f80cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 shutdown_connections 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 1 -- 172.21.15.138:0/2859032393 wait complete. 2024-03-29T11:22:26.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:26.351+0000 7f80d28e2640 1 librados: shutdown 2024-03-29T11:22:26.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:33.148 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.309+0000 7ff29d7d6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.309+0000 7ff29d7d6640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.309+0000 7ff29d7d6640 10 monclient: monmap: 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:33.310653+0000 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:33.310653+0000 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:33.287 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:33.288 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:33.288 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:33.288 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.311+0000 7ff29d7d6640 1 Processor -- start 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.311+0000 7ff29d7d6640 1 -- start start 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.311+0000 7ff29d7d6640 10 monclient: init 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff2980a9930 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff298149dc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:33.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff29814ff40 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:33.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980aae60 con 0x7ff298149dc0 2024-03-29T11:22:33.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980b4310 con 0x7ff29814ff40 2024-03-29T11:22:33.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980afdd0 con 0x7ff2980a9930 2024-03-29T11:22:33.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.312+0000 7ff29d7d6640 10 monclient(hunting): authenticate will time out at 974.621192s 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff296ffd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7ff2980a9930 0x7ff298143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:36560/0 (socket says 172.21.15.138:36560) 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff296ffd640 1 -- 172.21.15.138:0/3771508154 learned_addr learned my addr 172.21.15.138:0/3771508154 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071814565 0 0) 0x7ff2980afdd0 con 0x7ff2980a9930 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff295ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff295ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff295ffb640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.313+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280003520 con 0x7ff2980a9930 2024-03-29T11:22:33.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.314+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072047750 0 0) 0x7ff280003520 con 0x7ff2980a9930 2024-03-29T11:22:33.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.314+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 >> v1:172.21.15.152:6789/0 conn(0x7ff29814ff40 legacy=0x7ff298152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:33.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.315+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 >> v1:172.21.15.138:6789/0 conn(0x7ff298149dc0 legacy=0x7ff29814a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29814a950 con 0x7ff2980a9930 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff288001670 con 0x7ff2980a9930 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: dump: 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:33.295 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff29d7d6640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff29d7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2880018d0 con 0x7ff2980a9930 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 1 -- 172.21.15.138:0/3771508154 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff288001d20 con 0x7ff2980a9930 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff29d7d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:33.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff29d7d6640 4 set_mon_vals no callback set 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.316+0000 7ff295ffb640 10 monclient: dump: 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.317+0000 7ff29d7d6640 10 monclient: shutdown 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.317+0000 7ff29d7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.317+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3771508154 >> v1:172.21.15.138:6790/0 conn(0x7ff2980a9930 legacy=0x7ff298143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.317+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3771508154 shutdown_connections 2024-03-29T11:22:33.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.317+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3771508154 >> 172.21.15.138:0/3771508154 conn(0x7ff2980b35e0 msgr2=0x7ff2980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.318+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3771508154 shutdown_connections 2024-03-29T11:22:33.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.318+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3771508154 wait complete. 2024-03-29T11:22:33.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.319+0000 7ff29d7d6640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.319+0000 7ff29d7d6640 10 monclient: monmap: 2024-03-29T11:22:33.299 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:33.299 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:33.299 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:33.299 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:33.299 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.319+0000 7ff29d7d6640 1 librados: starting msgr at 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.319+0000 7ff29d7d6640 1 librados: starting objecter 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.320+0000 7ff29d7d6640 1 Processor -- start 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.320+0000 7ff29d7d6640 1 -- start start 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.320+0000 7ff29d7d6640 1 librados: setting wanted keys 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.320+0000 7ff29d7d6640 1 librados: calling monclient init 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.320+0000 7ff29d7d6640 10 monclient: init 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:33.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff298149dc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:33.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff29814ff40 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff29813e2e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2981db7c0 con 0x7ff298149dc0 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff298147740 con 0x7ff29813e2e0 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2981dc3d0 con 0x7ff29814ff40 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff29d7d6640 10 monclient(hunting): authenticate will time out at 974.630394s 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff2967fc640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7ff29814ff40 0x7ff2981489c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:36834/0 (socket says 172.21.15.138:36834) 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff2967fc640 1 -- 172.21.15.138:0/3859245292 learned_addr learned my addr 172.21.15.138:0/3859245292 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff27f7fe640 10 client.?.objecter ms_handle_connect 0x7ff29814ff40 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff27f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:33.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.321+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739616685 0 0) 0x7ff2981dc3d0 con 0x7ff29814ff40 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff284004eb0 con 0x7ff29814ff40 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2009059572 0 0) 0x7ff284004eb0 con 0x7ff29814ff40 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff284005300 con 0x7ff29814ff40 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff28c001920 con 0x7ff29814ff40 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:33.303 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.322+0000 7ff27f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 10 client.?.objecter ms_handle_connect 0x7ff298149dc0 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324602643 0 0) 0x7ff2981db7c0 con 0x7ff298149dc0 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:22:33.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.325+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2840058a0 con 0x7ff298149dc0 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 398260729 0 0) 0x7ff2840058a0 con 0x7ff298149dc0 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2981db7c0 con 0x7ff298149dc0 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff288001630 con 0x7ff298149dc0 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:33.308 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1409693835 0 0) 0x7ff284005300 con 0x7ff29814ff40 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 >> v1:172.21.15.152:6789/0 conn(0x7ff29813e2e0 legacy=0x7ff29813a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 >> v1:172.21.15.138:6789/0 conn(0x7ff298149dc0 legacy=0x7ff2981dbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 monclient: found mon.c 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2981dc560 con 0x7ff29814ff40 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:22:33.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 10 monclient: _renew_subs 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2981490d0 con 0x7ff29814ff40 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 10 monclient: _renew_subs 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29813abb0 con 0x7ff29814ff40 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff29d7d6640 1 librados: init done 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff28c001c40 con 0x7ff29814ff40 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff28c001920 con 0x7ff29814ff40 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:33.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff27f7fe640 10 monclient: dump: 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.326+0000 7ff2957fa640 4 set_mon_vals no callback set 2024-03-29T11:22:33.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.327+0000 7ff29d7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.327+0000 7ff29d7d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.327+0000 7ff29d7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.327+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff278005080 con 0x7ff29814ff40 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.328+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff28c0038a0 con 0x7ff29814ff40 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.329+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ff28c084600 con 0x7ff29814ff40 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.329+0000 7ff27f7fe640 10 client.4331.objecter ms_dispatch 0x7ff298000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.329+0000 7ff27f7fe640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.329+0000 7ff27f7fe640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.329+0000 7ff27f7fe640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.329+0000 7ff296ffd640 10 monclient: get_auth_request con 0x7ff28406e360 auth_method 0 2024-03-29T11:22:33.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.330+0000 7ff27f7fe640 10 client.4331.objecter ms_handle_connect 0x7ff28406e360 2024-03-29T11:22:33.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.338+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff28c04ff40 con 0x7ff29814ff40 2024-03-29T11:22:33.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.338+0000 7ff27f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.338+0000 7ff27f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:33.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.649+0000 7ff29d7d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.649+0000 7ff29d7d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.649+0000 7ff29d7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:33.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.649+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff278005330 con 0x7ff29814ff40 2024-03-29T11:22:33.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.650+0000 7ff27f7fe640 1 -- 172.21.15.138:0/3859245292 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ff28c04ff40 con 0x7ff29814ff40 2024-03-29T11:22:33.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.650+0000 7ff27f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.650+0000 7ff27f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:33.627 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:33.628 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:33.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.654+0000 7ff29d7d6640 10 librados: watch_flush enter 2024-03-29T11:22:33.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.656+0000 7ff29d7d6640 10 librados: watch_flush exit 2024-03-29T11:22:33.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.656+0000 7ff29d7d6640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.656+0000 7ff29d7d6640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.656+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff28406e360 legacy=0x7ff284070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.656+0000 7ff29d7d6640 10 monclient: shutdown 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.656+0000 7ff29d7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.656+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 >> v1:172.21.15.138:6790/0 conn(0x7ff29814ff40 legacy=0x7ff2981489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.657+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 shutdown_connections 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.657+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 >> 172.21.15.138:0/3859245292 conn(0x7ff2980b35e0 msgr2=0x7ff2980b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.657+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 shutdown_connections 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.657+0000 7ff29d7d6640 1 -- 172.21.15.138:0/3859245292 wait complete. 2024-03-29T11:22:33.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:33.657+0000 7ff29d7d6640 1 librados: shutdown 2024-03-29T11:22:33.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:40.458 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:40.596 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.619+0000 7f0b178f1640 10 monclient: get_monmap_and_config 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.619+0000 7f0b178f1640 10 monclient: build_initial_monmap 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.619+0000 7f0b178f1640 10 monclient: monmap: 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:40.621095+0000 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:40.621095+0000 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:40.597 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:40.599 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.621+0000 7f0b178f1640 1 Processor -- start 2024-03-29T11:22:40.599 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.621+0000 7f0b178f1640 1 -- start start 2024-03-29T11:22:40.599 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.621+0000 7f0b178f1640 10 monclient: init 2024-03-29T11:22:40.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:40.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 10 monclient(hunting): picked mon.noname-c con 0x7f0b1006ade0 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:40.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 10 monclient(hunting): picked mon.noname-b con 0x7f0b1006ed60 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 10 monclient(hunting): picked mon.noname-a con 0x7f0b10072340 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1006ff50 con 0x7f0b10072340 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b10070180 con 0x7f0b1006ade0 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1006f310 con 0x7f0b1006ed60 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.622+0000 7f0b178f1640 10 monclient(hunting): authenticate will time out at 981.931663s 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.623+0000 7f0b14e8c640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f0b1006ed60 0x7f0b10071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:60662/0 (socket says 172.21.15.138:60662) 2024-03-29T11:22:40.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.623+0000 7f0b14e8c640 1 -- 172.21.15.138:0/676105424 learned_addr learned my addr 172.21.15.138:0/676105424 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885344498 0 0) 0x7f0b1006f310 con 0x7f0b1006ed60 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0afc003520 con 0x7f0b1006ed60 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620542449 0 0) 0x7f0afc003520 con 0x7f0b1006ed60 2024-03-29T11:22:40.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.624+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 >> v1:172.21.15.152:6789/0 conn(0x7f0b1006ade0 legacy=0x7f0b101499e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 >> v1:172.21.15.138:6789/0 conn(0x7f0b10072340 legacy=0x7f0b10151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 1 monclient: found mon.noname-b 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100aeed0 con 0x7f0b1006ed60 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0b0c001690 con 0x7f0b1006ed60 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:40.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b07fff640 10 monclient: dump: 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b178f1640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.626+0000 7f0b178f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:40.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b0c001860 con 0x7f0b1006ed60 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b07fff640 1 -- 172.21.15.138:0/676105424 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0b0c001cb0 con 0x7f0b1006ed60 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b07fff640 10 monclient: dump: 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.606 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.607 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:40.607 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:40.607 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:40.607 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:40.607 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:40.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b178f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:40.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.627+0000 7f0b178f1640 4 set_mon_vals no callback set 2024-03-29T11:22:40.610 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.628+0000 7f0b178f1640 10 monclient: shutdown 2024-03-29T11:22:40.610 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.628+0000 7f0b178f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.610 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.628+0000 7f0b178f1640 1 -- 172.21.15.138:0/676105424 >> v1:172.21.15.138:6790/0 conn(0x7f0b1006ed60 legacy=0x7f0b10071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.628+0000 7f0b178f1640 1 -- 172.21.15.138:0/676105424 shutdown_connections 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.628+0000 7f0b178f1640 1 -- 172.21.15.138:0/676105424 >> 172.21.15.138:0/676105424 conn(0x7f0b100b4930 msgr2=0x7f0b100b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.628+0000 7f0b178f1640 1 -- 172.21.15.138:0/676105424 shutdown_connections 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.628+0000 7f0b178f1640 1 -- 172.21.15.138:0/676105424 wait complete. 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.629+0000 7f0b178f1640 10 monclient: build_initial_monmap 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.629+0000 7f0b178f1640 10 monclient: monmap: 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:40.611 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.630+0000 7f0b178f1640 1 librados: starting msgr at 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.630+0000 7f0b178f1640 1 librados: starting objecter 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.630+0000 7f0b178f1640 1 Processor -- start 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.630+0000 7f0b178f1640 1 -- start start 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.630+0000 7f0b178f1640 1 librados: setting wanted keys 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.630+0000 7f0b178f1640 1 librados: calling monclient init 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.630+0000 7f0b178f1640 10 monclient: init 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 10 monclient(hunting): picked mon.noname-b con 0x7f0b1006ade0 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 10 monclient(hunting): picked mon.noname-a con 0x7f0b1006ed60 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:40.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 10 monclient(hunting): picked mon.noname-c con 0x7f0b10072340 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b101db390 con 0x7f0b1006ed60 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b100a9e60 con 0x7f0b1006ade0 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b101472f0 con 0x7f0b10072340 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b178f1640 10 monclient(hunting): authenticate will time out at 981.940353s 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b15e8e640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f0b10072340 0x7f0b10141230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:34746/0 (socket says 172.21.15.138:34746) 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b15e8e640 1 -- 172.21.15.138:0/2541066580 learned_addr learned my addr 172.21.15.138:0/2541066580 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 10 client.?.objecter ms_handle_connect 0x7f0b10072340 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251158525 0 0) 0x7f0b101472f0 con 0x7f0b10072340 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af4004eb0 con 0x7f0b10072340 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.631+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 850235444 0 0) 0x7f0af4004eb0 con 0x7f0b10072340 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b101472f0 con 0x7f0b10072340 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0b08001ad0 con 0x7f0b10072340 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:40.614 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:40.615 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:40.615 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:40.615 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:40.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:40.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1502570629 0 0) 0x7f0b101472f0 con 0x7f0b10072340 2024-03-29T11:22:40.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.632+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 >> v1:172.21.15.152:6789/0 conn(0x7f0b1006ade0 legacy=0x7f0b100a9750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 >> v1:172.21.15.138:6789/0 conn(0x7f0b1006ed60 legacy=0x7f0b101db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 1 monclient: found mon.c 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b101dbfa0 con 0x7f0b10072340 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 10 client.?.objecter ms_handle_connect 0x7f0b1006ade0 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.635+0000 7f0b05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 10 monclient: _renew_subs 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b101449a0 con 0x7f0b10072340 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 10 monclient: _renew_subs 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b100aa780 con 0x7f0b10072340 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b178f1640 1 librados: init done 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b08001d30 con 0x7f0b10072340 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0b08004dc0 con 0x7f0b10072340 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b05ffb640 10 monclient: dump: 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.636+0000 7f0b077fe640 4 set_mon_vals no callback set 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.637+0000 7f0b178f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.637+0000 7f0b178f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.637+0000 7f0b178f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.637+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af0005080 con 0x7f0b10072340 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.638+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f0b08004f20 con 0x7f0b10072340 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.639+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f0b08084dd0 con 0x7f0b10072340 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.639+0000 7f0b05ffb640 10 client.4337.objecter ms_dispatch 0x7f0b10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.639+0000 7f0b05ffb640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.639+0000 7f0b05ffb640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.639+0000 7f0b05ffb640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:22:40.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.640+0000 7f0b1568d640 10 monclient: get_auth_request con 0x7f0af406db80 auth_method 0 2024-03-29T11:22:40.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.641+0000 7f0b05ffb640 10 client.4337.objecter ms_handle_connect 0x7f0af406db80 2024-03-29T11:22:40.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.646+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b080504e0 con 0x7f0b10072340 2024-03-29T11:22:40.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.646+0000 7f0b05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.646+0000 7f0b05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:40.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.963+0000 7f0b178f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.963+0000 7f0b178f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.963+0000 7f0b178f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:40.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.963+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0af0005330 con 0x7f0b10072340 2024-03-29T11:22:40.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.963+0000 7f0b05ffb640 1 -- 172.21.15.138:0/2541066580 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f0b080525e0 con 0x7f0b10072340 2024-03-29T11:22:40.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.964+0000 7f0b05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.964+0000 7f0b05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:40.941 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:40.941 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.967+0000 7f0b178f1640 10 librados: watch_flush enter 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.967+0000 7f0b178f1640 10 librados: watch_flush exit 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.967+0000 7f0b178f1640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.967+0000 7f0b178f1640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 >> v1:172.21.15.152:6816/3052585572 conn(0x7f0af406db80 legacy=0x7f0af4070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 10 monclient: shutdown 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 >> v1:172.21.15.138:6790/0 conn(0x7f0b10072340 legacy=0x7f0b10141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 shutdown_connections 2024-03-29T11:22:40.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 >> 172.21.15.138:0/2541066580 conn(0x7f0b100b4930 msgr2=0x7f0b100afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 shutdown_connections 2024-03-29T11:22:40.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 1 -- 172.21.15.138:0/2541066580 wait complete. 2024-03-29T11:22:40.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:40.968+0000 7f0b178f1640 1 librados: shutdown 2024-03-29T11:22:40.961 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:47.768 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.938+0000 7fa1a2c8d640 10 monclient: get_monmap_and_config 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.938+0000 7fa1a2c8d640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.938+0000 7fa1a2c8d640 10 monclient: monmap: 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:47.939763+0000 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:47.939763+0000 2024-03-29T11:22:47.918 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.940+0000 7fa1a2c8d640 1 Processor -- start 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.940+0000 7fa1a2c8d640 1 -- start start 2024-03-29T11:22:47.919 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.940+0000 7fa1a2c8d640 10 monclient: init 2024-03-29T11:22:47.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:47.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c0a9930 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:47.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c149dc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:47.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c14ff40 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:47.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c0aae60 con 0x7fa19c149dc0 2024-03-29T11:22:47.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c0b4310 con 0x7fa19c14ff40 2024-03-29T11:22:47.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c0afdd0 con 0x7fa19c0a9930 2024-03-29T11:22:47.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.941+0000 7fa1a2c8d640 10 monclient(hunting): authenticate will time out at 989.250543s 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.949+0000 7fa1a122a640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fa19c14ff40 0x7fa19c152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:34293/0 (socket says 172.21.15.138:34293) 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.949+0000 7fa1a122a640 1 -- 172.21.15.138:0/2813710960 learned_addr learned my addr 172.21.15.138:0/2813710960 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.949+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036409704 0 0) 0x7fa19c0aae60 con 0x7fa19c149dc0 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.949+0000 7fa19b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.949+0000 7fa19b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.949+0000 7fa19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.949+0000 7fa19b7fe640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.950+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa184003520 con 0x7fa19c149dc0 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.950+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3625167083 0 0) 0x7fa184003520 con 0x7fa19c149dc0 2024-03-29T11:22:47.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.950+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 >> v1:172.21.15.138:6790/0 conn(0x7fa19c0a9930 legacy=0x7fa19c143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:47.928 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 >> v1:172.21.15.152:6789/0 conn(0x7fa19c14ff40 legacy=0x7fa19c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.928 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c14a950 con 0x7fa19c149dc0 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa190001690 con 0x7fa19c149dc0 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa19b7fe640 10 monclient: dump: 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:47.929 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa1a2c8d640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.951+0000 7fa1a2c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.952+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa190001860 con 0x7fa19c149dc0 2024-03-29T11:22:47.930 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.952+0000 7fa19b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.952+0000 7fa19b7fe640 1 -- 172.21.15.138:0/2813710960 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa190001cb0 con 0x7fa19c149dc0 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.952+0000 7fa19b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.952+0000 7fa19b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.952+0000 7fa19b7fe640 10 monclient: dump: 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:47.931 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:47.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.954+0000 7fa1a2c8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:47.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.954+0000 7fa1a2c8d640 4 set_mon_vals no callback set 2024-03-29T11:22:47.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.955+0000 7fa1a2c8d640 10 monclient: shutdown 2024-03-29T11:22:47.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.955+0000 7fa1a2c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:47.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.955+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/2813710960 >> v1:172.21.15.138:6789/0 conn(0x7fa19c149dc0 legacy=0x7fa19c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.955+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/2813710960 shutdown_connections 2024-03-29T11:22:47.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.955+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/2813710960 >> 172.21.15.138:0/2813710960 conn(0x7fa19c0b35e0 msgr2=0x7fa19c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:47.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.956+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/2813710960 shutdown_connections 2024-03-29T11:22:47.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.957+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/2813710960 wait complete. 2024-03-29T11:22:47.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.958+0000 7fa1a2c8d640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.958+0000 7fa1a2c8d640 10 monclient: monmap: 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.959+0000 7fa1a2c8d640 1 librados: starting msgr at 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.959+0000 7fa1a2c8d640 1 librados: starting objecter 2024-03-29T11:22:47.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.959+0000 7fa1a2c8d640 1 Processor -- start 2024-03-29T11:22:47.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.961+0000 7fa1a2c8d640 1 -- start start 2024-03-29T11:22:47.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.961+0000 7fa1a2c8d640 1 librados: setting wanted keys 2024-03-29T11:22:47.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.961+0000 7fa1a2c8d640 1 librados: calling monclient init 2024-03-29T11:22:47.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.961+0000 7fa1a2c8d640 10 monclient: init 2024-03-29T11:22:47.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.962+0000 7fa1a2c8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.962+0000 7fa1a2c8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:47.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.962+0000 7fa1a2c8d640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c0a9930 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:47.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.964+0000 7fa1a2c8d640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c14ff40 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:47.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.964+0000 7fa1a2c8d640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c0ab380 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:47.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.964+0000 7fa1a2c8d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c149ad0 con 0x7fa19c0ab380 2024-03-29T11:22:47.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.964+0000 7fa1a2c8d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c1db850 con 0x7fa19c14ff40 2024-03-29T11:22:47.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.964+0000 7fa1a2c8d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c148260 con 0x7fa19c0a9930 2024-03-29T11:22:47.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.964+0000 7fa1a2c8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.964+0000 7fa1a2c8d640 10 monclient(hunting): authenticate will time out at 989.273825s 2024-03-29T11:22:47.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.969+0000 7fa1a122a640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fa19c0ab380 0x7fa19c148d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:59115/0 (socket says 172.21.15.138:59115) 2024-03-29T11:22:47.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.969+0000 7fa1a122a640 1 -- 172.21.15.138:0/440790341 learned_addr learned my addr 172.21.15.138:0/440790341 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:47.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.971+0000 7fa1997fa640 10 client.?.objecter ms_handle_connect 0x7fa19c0ab380 2024-03-29T11:22:47.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.971+0000 7fa1997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 client.?.objecter ms_handle_connect 0x7fa19c0a9930 2024-03-29T11:22:47.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073282495 0 0) 0x7fa19c149ad0 con 0x7fa19c0ab380 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa188004eb0 con 0x7fa19c0ab380 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325451493 0 0) 0x7fa19c148260 con 0x7fa19c0a9930 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c149ad0 con 0x7fa19c0a9930 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2723974944 0 0) 0x7fa188004eb0 con 0x7fa19c0ab380 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.972+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1880064f0 con 0x7fa19c0ab380 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 client.?.objecter ms_handle_connect 0x7fa19c14ff40 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314523893 0 0) 0x7fa19c1db850 con 0x7fa19c14ff40 2024-03-29T11:22:47.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa188006930 con 0x7fa19c14ff40 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4058103517 0 0) 0x7fa19c149ad0 con 0x7fa19c0a9930 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa19c1db850 con 0x7fa19c0a9930 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa18c0017b0 con 0x7fa19c0a9930 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): dump: 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.954 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa194001bd0 con 0x7fa19c0ab380 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient(hunting): dump: 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:47.955 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3935576307 0 0) 0x7fa1880064f0 con 0x7fa19c0ab380 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 >> v1:172.21.15.138:6790/0 conn(0x7fa19c0a9930 legacy=0x7fa19c1dbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 >> v1:172.21.15.152:6789/0 conn(0x7fa19c14ff40 legacy=0x7fa19c13a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 monclient: found mon.a 2024-03-29T11:22:47.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c1dc480 con 0x7fa19c0ab380 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.973+0000 7fa1997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 10 monclient: _renew_subs 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19c13ab60 con 0x7fa19c0ab380 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 10 monclient: _renew_subs 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:47.960 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19c149630 con 0x7fa19c0ab380 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.974+0000 7fa1a2c8d640 1 librados: init done 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1940020c0 con 0x7fa19c0ab380 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa194001bd0 con 0x7fa19c0ab380 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1997fa640 10 monclient: dump: 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:47.961 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fa19401ab10 con 0x7fa19c0ab380 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1a2c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1a2c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1a2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.976+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19c075f10 con 0x7fa19c0ab380 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.977+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fa194084b20 con 0x7fa19c0ab380 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.977+0000 7fa1997fa640 10 client.4335.objecter ms_dispatch 0x7fa19c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.977+0000 7fa1997fa640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.977+0000 7fa1997fa640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.977+0000 7fa1997fa640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.977+0000 7fa19affd640 4 set_mon_vals no callback set 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.978+0000 7fa1a0a29640 10 monclient: get_auth_request con 0x7fa18806e5d0 auth_method 0 2024-03-29T11:22:47.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.979+0000 7fa1997fa640 10 client.4335.objecter ms_handle_connect 0x7fa18806e5d0 2024-03-29T11:22:47.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.986+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1940502b0 con 0x7fa19c0ab380 2024-03-29T11:22:47.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.986+0000 7fa1997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:47.986+0000 7fa1997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:48.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.304+0000 7fa1a2c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.304+0000 7fa1a2c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.282 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.304+0000 7fa1a2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:22:48.282 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.304+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa19c1dc610 con 0x7fa19c0ab380 2024-03-29T11:22:48.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.306+0000 7fa1997fa640 1 -- 172.21.15.138:0/440790341 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fa1940523b0 con 0x7fa19c0ab380 2024-03-29T11:22:48.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.306+0000 7fa1997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.306+0000 7fa1997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:48.286 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:48.286 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:48.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.312+0000 7fa1a2c8d640 10 librados: watch_flush enter 2024-03-29T11:22:48.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.314+0000 7fa1a2c8d640 10 librados: watch_flush exit 2024-03-29T11:22:48.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.314+0000 7fa1a2c8d640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-29T11:22:48.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.314+0000 7fa1a2c8d640 10 client.4335.objecter successfully canceled tick 2024-03-29T11:22:48.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.314+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 >> v1:172.21.15.152:6816/3052585572 conn(0x7fa18806e5d0 legacy=0x7fa188070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.314+0000 7fa1a2c8d640 10 monclient: shutdown 2024-03-29T11:22:48.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.314+0000 7fa1a2c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:48.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.314+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 >> v1:172.21.15.138:6789/0 conn(0x7fa19c0ab380 legacy=0x7fa19c148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.293 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.316+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 shutdown_connections 2024-03-29T11:22:48.293 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.316+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 >> 172.21.15.138:0/440790341 conn(0x7fa19c0b35e0 msgr2=0x7fa19c143800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:48.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.319+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 shutdown_connections 2024-03-29T11:22:48.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.319+0000 7fa1a2c8d640 1 -- 172.21.15.138:0/440790341 wait complete. 2024-03-29T11:22:48.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:48.320+0000 7fa1a2c8d640 1 librados: shutdown 2024-03-29T11:22:48.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:55.122 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.291+0000 7f396d15d640 10 monclient: get_monmap_and_config 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.291+0000 7f396d15d640 10 monclient: build_initial_monmap 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.291+0000 7f396d15d640 10 monclient: monmap: 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:22:55.292531+0000 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:22:55.292531+0000 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:55.269 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:55.270 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:22:55.270 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:22:55.270 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:55.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.294+0000 7f396d15d640 1 Processor -- start 2024-03-29T11:22:55.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 1 -- start start 2024-03-29T11:22:55.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 10 monclient: init 2024-03-29T11:22:55.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:55.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:55.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 10 monclient(hunting): picked mon.noname-b con 0x7f39680b0770 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:55.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 10 monclient(hunting): picked mon.noname-c con 0x7f396814a700 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:55.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 10 monclient(hunting): picked mon.noname-a con 0x7f39680ac5a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:55.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.295+0000 7f396d15d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680aba70 con 0x7f39680ac5a0 2024-03-29T11:22:55.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.296+0000 7f396d15d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680aecb0 con 0x7f396814a700 2024-03-29T11:22:55.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.296+0000 7f396d15d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680af150 con 0x7f39680b0770 2024-03-29T11:22:55.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.296+0000 7f396d15d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:55.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.296+0000 7f396d15d640 10 monclient(hunting): authenticate will time out at 996.605080s 2024-03-29T11:22:55.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.301+0000 7f3966d9d640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f39680b0770 0x7f3968149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:59517/0 (socket says 172.21.15.138:59517) 2024-03-29T11:22:55.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.301+0000 7f3966d9d640 1 -- 172.21.15.138:0/4092749820 learned_addr learned my addr 172.21.15.138:0/4092749820 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:55.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502655481 0 0) 0x7f39680aba70 con 0x7f39680ac5a0 2024-03-29T11:22:55.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:22:55.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3950003520 con 0x7f39680ac5a0 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1513196629 0 0) 0x7f3950003520 con 0x7f39680ac5a0 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 >> v1:172.21.15.138:6790/0 conn(0x7f39680b0770 legacy=0x7f3968149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 >> v1:172.21.15.152:6789/0 conn(0x7f396814a700 legacy=0x7f39680abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 monclient: found mon.noname-a 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39680ac9d0 con 0x7f39680ac5a0 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient: _finish_auth 0 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f396d15d640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f396d15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:55.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f3958001690 con 0x7f39680ac5a0 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f3965d9b640 10 monclient: dump: 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.302+0000 7f396d15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3958001860 con 0x7f39680ac5a0 2024-03-29T11:22:55.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f3965d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f396d15d640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f396d15d640 4 set_mon_vals no callback set 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f3965d9b640 1 -- 172.21.15.138:0/4092749820 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f3958001c70 con 0x7f39680ac5a0 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f3965d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f3965d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f3965d9b640 10 monclient: dump: 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:55.290 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f396d15d640 10 monclient: shutdown 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f396d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.303+0000 7f396d15d640 1 -- 172.21.15.138:0/4092749820 >> v1:172.21.15.138:6789/0 conn(0x7f39680ac5a0 legacy=0x7f3968151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.304+0000 7f396d15d640 1 -- 172.21.15.138:0/4092749820 shutdown_connections 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.304+0000 7f396d15d640 1 -- 172.21.15.138:0/4092749820 >> 172.21.15.138:0/4092749820 conn(0x7f396806b660 msgr2=0x7f396806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.304+0000 7f396d15d640 1 -- 172.21.15.138:0/4092749820 shutdown_connections 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.304+0000 7f396d15d640 1 -- 172.21.15.138:0/4092749820 wait complete. 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.305+0000 7f396d15d640 10 monclient: build_initial_monmap 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.305+0000 7f396d15d640 10 monclient: monmap: 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:55.291 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.305+0000 7f396d15d640 1 librados: starting msgr at 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.305+0000 7f396d15d640 1 librados: starting objecter 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 1 Processor -- start 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 1 -- start start 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 1 librados: setting wanted keys 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 1 librados: calling monclient init 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 10 monclient: init 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:55.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.306+0000 7f396d15d640 10 monclient(hunting): picked mon.noname-a con 0x7f39680ac5a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:22:55.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.308+0000 7f396d15d640 10 monclient(hunting): picked mon.noname-b con 0x7f39680b0770 addr v1:172.21.15.152:6789/0 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396659c640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f39680b0770 0x7f39681f7870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:40937/0 (socket says 172.21.15.138:40937) 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396d15d640 10 monclient(hunting): picked mon.noname-c con 0x7f396814c2b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396d15d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396814fb10 con 0x7f39680ac5a0 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396d15d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396814fd80 con 0x7f39680b0770 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396d15d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396814e250 con 0x7f396814c2b0 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396d15d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396d15d640 10 monclient(hunting): authenticate will time out at 996.619609s 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.310+0000 7f396659c640 1 -- 172.21.15.138:0/3776167340 learned_addr learned my addr 172.21.15.138:0/3776167340 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.312+0000 7f394f7fe640 10 client.?.objecter ms_handle_connect 0x7f39680b0770 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.312+0000 7f394f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.312+0000 7f394f7fe640 10 client.?.objecter ms_handle_connect 0x7f39680ac5a0 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.312+0000 7f394f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830666868 0 0) 0x7f396814fd80 con 0x7f39680b0770 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3954004eb0 con 0x7f39680b0770 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541789504 0 0) 0x7f396814fb10 con 0x7f39680ac5a0 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f396814fd80 con 0x7f39680ac5a0 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 client.?.objecter ms_handle_connect 0x7f396814c2b0 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.314+0000 7f394f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231247908 0 0) 0x7f396814e250 con 0x7f396814c2b0 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f396814fb10 con 0x7f396814c2b0 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1278340549 0 0) 0x7f396814fb10 con 0x7f396814c2b0 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f396814e250 con 0x7f396814c2b0 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f3958001b70 con 0x7f396814c2b0 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.299 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3976574950 0 0) 0x7f396814e250 con 0x7f396814c2b0 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 >> v1:172.21.15.152:6789/0 conn(0x7f39680b0770 legacy=0x7f39681f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 >> v1:172.21.15.138:6789/0 conn(0x7f39680ac5a0 legacy=0x7f396814bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 monclient: found mon.c 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396814eaa0 con 0x7f396814c2b0 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:55.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3958001cd0 con 0x7f396814c2b0 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f3958003b00 con 0x7f396814c2b0 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.315+0000 7f394f7fe640 10 monclient: dump: 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:22:55.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 10 monclient: _renew_subs 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39681f7f80 con 0x7f396814c2b0 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 10 monclient: _renew_subs 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f396814d6b0 con 0x7f396814c2b0 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.317+0000 7f396d15d640 1 librados: init done 2024-03-29T11:22:55.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.318+0000 7f396559a640 4 set_mon_vals no callback set 2024-03-29T11:22:55.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.319+0000 7f396d15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.319+0000 7f396d15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.319+0000 7f396d15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.319+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3948005080 con 0x7f396814c2b0 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.321+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f3958019ad0 con 0x7f396814c2b0 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.321+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f3958083840 con 0x7f396814c2b0 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.321+0000 7f394f7fe640 10 client.4346.objecter ms_dispatch 0x7f3968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.321+0000 7f394f7fe640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.321+0000 7f394f7fe640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.321+0000 7f394f7fe640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.322+0000 7f3966d9d640 10 monclient: get_auth_request con 0x7f395406dcf0 auth_method 0 2024-03-29T11:22:55.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.323+0000 7f394f7fe640 10 client.4346.objecter ms_handle_connect 0x7f395406dcf0 2024-03-29T11:22:55.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.331+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f395804f120 con 0x7f396814c2b0 2024-03-29T11:22:55.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.331+0000 7f394f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.331+0000 7f394f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:55.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.655+0000 7f396d15d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.655+0000 7f396d15d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.655+0000 7f396d15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:22:55.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.655+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3948005330 con 0x7f396814c2b0 2024-03-29T11:22:55.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.657+0000 7f394f7fe640 1 -- 172.21.15.138:0/3776167340 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f3958051050 con 0x7f396814c2b0 2024-03-29T11:22:55.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.657+0000 7f394f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.657+0000 7f394f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:55.636 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:22:55.636 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 10 librados: watch_flush enter 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 10 librados: watch_flush exit 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 10 client.4346.objecter successfully canceled tick 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 >> v1:172.21.15.152:6816/3052585572 conn(0x7f395406dcf0 legacy=0x7f39540701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 10 monclient: shutdown 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 >> v1:172.21.15.138:6790/0 conn(0x7f396814c2b0 legacy=0x7f3968200850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 shutdown_connections 2024-03-29T11:22:55.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.661+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 >> 172.21.15.138:0/3776167340 conn(0x7f396806b660 msgr2=0x7f39680b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.664+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 shutdown_connections 2024-03-29T11:22:55.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.664+0000 7f396d15d640 1 -- 172.21.15.138:0/3776167340 wait complete. 2024-03-29T11:22:55.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:22:55.664+0000 7f396d15d640 1 librados: shutdown 2024-03-29T11:22:55.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:02.465 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:02.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.642+0000 7f1c7e5c3640 10 monclient: get_monmap_and_config 2024-03-29T11:23:02.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.642+0000 7f1c7e5c3640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.642+0000 7f1c7e5c3640 10 monclient: monmap: 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:02.643569+0000 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:02.643569+0000 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:02.621 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:02.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.644+0000 7f1c7e5c3640 1 Processor -- start 2024-03-29T11:23:02.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.644+0000 7f1c7e5c3640 1 -- start start 2024-03-29T11:23:02.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.644+0000 7f1c7e5c3640 10 monclient: init 2024-03-29T11:23:02.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:02.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 10 monclient(hunting): picked mon.noname-b con 0x7f1c7814a400 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:02.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 10 monclient(hunting): picked mon.noname-a con 0x7f1c780aa500 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:02.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 10 monclient(hunting): picked mon.noname-c con 0x7f1c78149800 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:02.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c780aca50 con 0x7f1c780aa500 2024-03-29T11:23:02.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c78149cb0 con 0x7f1c78149800 2024-03-29T11:23:02.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c780acc80 con 0x7f1c7814a400 2024-03-29T11:23:02.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.645+0000 7f1c7e5c3640 10 monclient(hunting): authenticate will time out at 1003.954520s 2024-03-29T11:23:02.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.651+0000 7f1c7ddc2640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f1c78149800 0x7f1c78151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:52601/0 (socket says 172.21.15.138:52601) 2024-03-29T11:23:02.630 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.652+0000 7f1c7ddc2640 1 -- 172.21.15.138:0/4064652178 learned_addr learned my addr 172.21.15.138:0/4064652178 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52685298 0 0) 0x7f1c780aca50 con 0x7f1c780aa500 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c64003520 con 0x7f1c780aa500 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920719852 0 0) 0x7f1c78149cb0 con 0x7f1c78149800 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:23:02.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.655+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c780aca50 con 0x7f1c78149800 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2878141425 0 0) 0x7f1c64003520 con 0x7f1c780aa500 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 >> v1:172.21.15.138:6790/0 conn(0x7f1c7814a400 legacy=0x7f1c780aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 >> v1:172.21.15.152:6789/0 conn(0x7f1c78149800 legacy=0x7f1c78151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 1 monclient: found mon.noname-a 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c780ac350 con 0x7f1c780aa500 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c7e5c3640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c7e5c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1c680015c0 con 0x7f1c780aa500 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c6ffff640 10 monclient: dump: 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:02.636 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:02.637 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:02.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.656+0000 7f1c7e5c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.660+0000 7f1c6ffff640 1 -- 172.21.15.138:0/4064652178 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c68001720 con 0x7f1c780aa500 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.660+0000 7f1c6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.660+0000 7f1c7e5c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.660+0000 7f1c7e5c3640 4 set_mon_vals no callback set 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.661+0000 7f1c7e5c3640 10 monclient: shutdown 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.661+0000 7f1c7e5c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.661+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/4064652178 >> v1:172.21.15.138:6789/0 conn(0x7f1c780aa500 legacy=0x7f1c780ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.661+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/4064652178 shutdown_connections 2024-03-29T11:23:02.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.661+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/4064652178 >> 172.21.15.138:0/4064652178 conn(0x7f1c780b3a30 msgr2=0x7f1c780b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.662+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/4064652178 shutdown_connections 2024-03-29T11:23:02.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.663+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/4064652178 wait complete. 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.663+0000 7f1c7e5c3640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.663+0000 7f1c7e5c3640 10 monclient: monmap: 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:02.642 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:02.643 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:02.643 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.664+0000 7f1c7e5c3640 1 librados: starting msgr at 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.664+0000 7f1c7e5c3640 1 librados: starting objecter 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.664+0000 7f1c7e5c3640 1 Processor -- start 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.664+0000 7f1c7e5c3640 1 -- start start 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.664+0000 7f1c7e5c3640 1 librados: setting wanted keys 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.664+0000 7f1c7e5c3640 1 librados: calling monclient init 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.664+0000 7f1c7e5c3640 10 monclient: init 2024-03-29T11:23:02.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 10 monclient(hunting): picked mon.noname-b con 0x7f1c780aa500 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 10 monclient(hunting): picked mon.noname-c con 0x7f1c78149800 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 10 monclient(hunting): picked mon.noname-a con 0x7f1c7814a400 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c781487b0 con 0x7f1c7814a400 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c781476c0 con 0x7f1c780aa500 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c781472d0 con 0x7f1c78149800 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.665+0000 7f1c7e5c3640 10 monclient(hunting): authenticate will time out at 1003.974455s 2024-03-29T11:23:02.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.669+0000 7f1c7ddc2640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f1c7814a400 0x7f1c781448f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47571/0 (socket says 172.21.15.138:47571) 2024-03-29T11:23:02.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.669+0000 7f1c7ddc2640 1 -- 172.21.15.138:0/1076253705 learned_addr learned my addr 172.21.15.138:0/1076253705 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:02.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.670+0000 7f1c6dffb640 10 client.?.objecter ms_handle_connect 0x7f1c7814a400 2024-03-29T11:23:02.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.671+0000 7f1c6dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.671+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519386750 0 0) 0x7f1c781487b0 con 0x7f1c7814a400 2024-03-29T11:23:02.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.671+0000 7f1c6dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.671+0000 7f1c6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.671+0000 7f1c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.671+0000 7f1c6dffb640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:23:02.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.671+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c5c004eb0 con 0x7f1c7814a400 2024-03-29T11:23:02.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 client.?.objecter ms_handle_connect 0x7f1c780aa500 2024-03-29T11:23:02.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089855417 0 0) 0x7f1c781476c0 con 0x7f1c780aa500 2024-03-29T11:23:02.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c781487b0 con 0x7f1c780aa500 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3173851817 0 0) 0x7f1c5c004eb0 con 0x7f1c7814a400 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c781476c0 con 0x7f1c7814a400 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1c70001b70 con 0x7f1c7814a400 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): dump: 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:02.657 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 client.?.objecter ms_handle_connect 0x7f1c78149800 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.674+0000 7f1c6dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655136221 0 0) 0x7f1c781472d0 con 0x7f1c78149800 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:23:02.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c5c004eb0 con 0x7f1c78149800 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796125996 0 0) 0x7f1c781476c0 con 0x7f1c7814a400 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 >> v1:172.21.15.138:6790/0 conn(0x7f1c78149800 legacy=0x7f1c781441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 >> v1:172.21.15.152:6789/0 conn(0x7f1c780aa500 legacy=0x7f1c78146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 1 monclient: found mon.a 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c78147460 con 0x7f1c7814a400 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 10 monclient: _renew_subs 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:02.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c78071fc0 con 0x7f1c7814a400 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 10 monclient: _renew_subs 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c780722c0 con 0x7f1c7814a400 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.675+0000 7f1c7e5c3640 1 librados: init done 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.676+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c70002080 con 0x7f1c7814a400 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.676+0000 7f1c6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.676+0000 7f1c6f7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.676+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1c70004d00 con 0x7f1c7814a400 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.676+0000 7f1c6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.676+0000 7f1c6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.676+0000 7f1c6dffb640 10 monclient: dump: 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:02.660 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:02.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.678+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f1c70004e60 con 0x7f1c7814a400 2024-03-29T11:23:02.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.679+0000 7f1c7e5c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.679+0000 7f1c7e5c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.679+0000 7f1c7e5c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.679+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c58005080 con 0x7f1c7814a400 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.680+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f1c70084d50 con 0x7f1c7814a400 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.680+0000 7f1c6dffb640 10 client.4347.objecter ms_dispatch 0x7f1c78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.680+0000 7f1c6dffb640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.680+0000 7f1c6dffb640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.680+0000 7f1c6dffb640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.680+0000 7f1c7d5c1640 10 monclient: get_auth_request con 0x7f1c5c06de40 auth_method 0 2024-03-29T11:23:02.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.684+0000 7f1c6dffb640 10 client.4347.objecter ms_handle_connect 0x7f1c5c06de40 2024-03-29T11:23:02.668 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.690+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c70050630 con 0x7f1c7814a400 2024-03-29T11:23:02.671 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.691+0000 7f1c6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.671 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:02.691+0000 7f1c6dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:03.002 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.024+0000 7f1c7e5c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.024+0000 7f1c7e5c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.024+0000 7f1c7e5c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:03.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.024+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c58005330 con 0x7f1c7814a400 2024-03-29T11:23:03.007 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:03.007 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:03.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.029+0000 7f1c6dffb640 1 -- 172.21.15.138:0/1076253705 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f1c70052560 con 0x7f1c7814a400 2024-03-29T11:23:03.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.029+0000 7f1c6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.029+0000 7f1c6dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:03.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.032+0000 7f1c7e5c3640 10 librados: watch_flush enter 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.032+0000 7f1c7e5c3640 10 librados: watch_flush exit 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.032+0000 7f1c7e5c3640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.032+0000 7f1c7e5c3640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.033+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 >> v1:172.21.15.152:6816/3052585572 conn(0x7f1c5c06de40 legacy=0x7f1c5c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.033+0000 7f1c7e5c3640 10 monclient: shutdown 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.033+0000 7f1c7e5c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.033+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 >> v1:172.21.15.138:6789/0 conn(0x7f1c7814a400 legacy=0x7f1c781448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.033+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 shutdown_connections 2024-03-29T11:23:03.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.033+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 >> 172.21.15.138:0/1076253705 conn(0x7f1c780b3a30 msgr2=0x7f1c780b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:03.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.038+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 shutdown_connections 2024-03-29T11:23:03.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.038+0000 7f1c7e5c3640 1 -- 172.21.15.138:0/1076253705 wait complete. 2024-03-29T11:23:03.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:03.038+0000 7f1c7e5c3640 1 librados: shutdown 2024-03-29T11:23:03.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:09.839 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:10.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.021+0000 7ff6a920e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:10.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.021+0000 7ff6a920e640 10 monclient: build_initial_monmap 2024-03-29T11:23:10.000 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.021+0000 7ff6a920e640 10 monclient: monmap: 2024-03-29T11:23:10.000 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:10.000 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:10.000 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:10.022950+0000 2024-03-29T11:23:10.000 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:10.022950+0000 2024-03-29T11:23:10.001 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:10.001 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:10.001 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:10.001 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:10.001 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:10.001 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:10.002 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.022+0000 7ff6a920e640 1 Processor -- start 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.022+0000 7ff6a920e640 1 -- start start 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.022+0000 7ff6a920e640 10 monclient: init 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a40b1f60 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a4071eb0 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a4143d10 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a406ef80 con 0x7ff6a4143d10 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a406f260 con 0x7ff6a40b1f60 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a40b40c0 con 0x7ff6a4071eb0 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:10.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.023+0000 7ff6a920e640 10 monclient(hunting): authenticate will time out at 1011.332907s 2024-03-29T11:23:10.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.027+0000 7ff6a359e640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ff6a4143d10 0x7ff6a4146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:43926/0 (socket says 172.21.15.138:43926) 2024-03-29T11:23:10.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.027+0000 7ff6a359e640 1 -- 172.21.15.138:0/988744017 learned_addr learned my addr 172.21.15.138:0/988744017 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111668440 0 0) 0x7ff6a406ef80 con 0x7ff6a4143d10 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff68c003520 con 0x7ff6a4143d10 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224028534 0 0) 0x7ff6a40b40c0 con 0x7ff6a4071eb0 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:23:10.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a406ef80 con 0x7ff6a4071eb0 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.028+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251486256 0 0) 0x7ff68c003520 con 0x7ff6a4143d10 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 >> v1:172.21.15.138:6790/0 conn(0x7ff6a4071eb0 legacy=0x7ff6a4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 >> v1:172.21.15.152:6789/0 conn(0x7ff6a40b1f60 legacy=0x7ff6a4143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 1 monclient: found mon.noname-a 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a4155d70 con 0x7ff6a4143d10 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a920e640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a920e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff69c001a10 con 0x7ff6a4143d10 2024-03-29T11:23:10.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.029+0000 7ff6a1d9b640 10 monclient: dump: 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.030+0000 7ff6a920e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.030+0000 7ff6a1d9b640 1 -- 172.21.15.138:0/988744017 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff69c001c30 con 0x7ff6a4143d10 2024-03-29T11:23:10.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.030+0000 7ff6a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:10.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.030+0000 7ff6a920e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:10.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.030+0000 7ff6a920e640 4 set_mon_vals no callback set 2024-03-29T11:23:10.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.031+0000 7ff6a920e640 10 monclient: shutdown 2024-03-29T11:23:10.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.031+0000 7ff6a920e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:10.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.031+0000 7ff6a920e640 1 -- 172.21.15.138:0/988744017 >> v1:172.21.15.138:6789/0 conn(0x7ff6a4143d10 legacy=0x7ff6a4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.031+0000 7ff6a920e640 1 -- 172.21.15.138:0/988744017 shutdown_connections 2024-03-29T11:23:10.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.031+0000 7ff6a920e640 1 -- 172.21.15.138:0/988744017 >> 172.21.15.138:0/988744017 conn(0x7ff6a4066a00 msgr2=0x7ff6a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:10.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.032+0000 7ff6a920e640 1 -- 172.21.15.138:0/988744017 shutdown_connections 2024-03-29T11:23:10.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.032+0000 7ff6a920e640 1 -- 172.21.15.138:0/988744017 wait complete. 2024-03-29T11:23:10.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.033+0000 7ff6a920e640 10 monclient: build_initial_monmap 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.033+0000 7ff6a920e640 10 monclient: monmap: 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.034+0000 7ff6a920e640 1 librados: starting msgr at 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.034+0000 7ff6a920e640 1 librados: starting objecter 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.034+0000 7ff6a920e640 1 Processor -- start 2024-03-29T11:23:10.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.034+0000 7ff6a920e640 1 -- start start 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.034+0000 7ff6a920e640 1 librados: setting wanted keys 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.034+0000 7ff6a920e640 1 librados: calling monclient init 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.034+0000 7ff6a920e640 10 monclient: init 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a40b1f60 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a4143d10 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a40a9f70 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a41e81a0 con 0x7ff6a4143d10 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a40ac010 con 0x7ff6a40b1f60 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a40ac2f0 con 0x7ff6a40a9f70 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:10.017 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.035+0000 7ff6a920e640 10 monclient(hunting): authenticate will time out at 1011.344874s 2024-03-29T11:23:10.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.037+0000 7ff6a259c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ff6a4143d10 0x7ff6a41e7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:46308/0 (socket says 172.21.15.138:46308) 2024-03-29T11:23:10.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.037+0000 7ff6a259c640 1 -- 172.21.15.138:0/2195771465 learned_addr learned my addr 172.21.15.138:0/2195771465 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:10.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 10 client.?.objecter ms_handle_connect 0x7ff6a4143d10 2024-03-29T11:23:10.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869011501 0 0) 0x7ff6a41e81a0 con 0x7ff6a4143d10 2024-03-29T11:23:10.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.021 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.039+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff690004eb0 con 0x7ff6a4143d10 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 client.?.objecter ms_handle_connect 0x7ff6a40a9f70 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906375879 0 0) 0x7ff6a40ac2f0 con 0x7ff6a40a9f70 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a41e81a0 con 0x7ff6a40a9f70 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2097424110 0 0) 0x7ff690004eb0 con 0x7ff6a4143d10 2024-03-29T11:23:10.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6a40ac2f0 con 0x7ff6a4143d10 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff694001a90 con 0x7ff6a4143d10 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:10.023 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.040+0000 7ff68b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 10 client.?.objecter ms_handle_connect 0x7ff6a40b1f60 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3304673726 0 0) 0x7ff6a40ac2f0 con 0x7ff6a4143d10 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 >> v1:172.21.15.138:6790/0 conn(0x7ff6a40a9f70 legacy=0x7ff6a41e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 >> v1:172.21.15.152:6789/0 conn(0x7ff6a40b1f60 legacy=0x7ff6a40aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 1 monclient: found mon.a 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a40ac5d0 con 0x7ff6a4143d10 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff68b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 10 monclient: _renew_subs 2024-03-29T11:23:10.024 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:10.025 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a40aae60 con 0x7ff6a4143d10 2024-03-29T11:23:10.025 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:10.025 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 10 monclient: _renew_subs 2024-03-29T11:23:10.025 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:10.025 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a4146890 con 0x7ff6a4143d10 2024-03-29T11:23:10.025 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.041+0000 7ff6a920e640 1 librados: init done 2024-03-29T11:23:10.026 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.044+0000 7ff6a920e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.026 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.044+0000 7ff6a920e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.026 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.044+0000 7ff6a920e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:10.026 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.044+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a4075f10 con 0x7ff6a4143d10 2024-03-29T11:23:10.026 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.047+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff694001cd0 con 0x7ff6a4143d10 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.047+0000 7ff68b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.047+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff694003a60 con 0x7ff6a4143d10 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.047+0000 7ff6a0d99640 4 set_mon_vals no callback set 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.047+0000 7ff68b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.047+0000 7ff68b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.047+0000 7ff68b7fe640 10 monclient: dump: 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:10.027 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:10.028 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:10.028 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.048+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff694003c80 con 0x7ff6a4143d10 2024-03-29T11:23:10.030 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.051+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ff694084ae0 con 0x7ff6a4143d10 2024-03-29T11:23:10.030 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.051+0000 7ff68b7fe640 10 client.4353.objecter ms_dispatch 0x7ff6a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:10.030 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.051+0000 7ff68b7fe640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:10.030 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.051+0000 7ff68b7fe640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:10.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.052+0000 7ff68b7fe640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T11:23:10.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.052+0000 7ff6a2d9d640 10 monclient: get_auth_request con 0x7ff69006dd20 auth_method 0 2024-03-29T11:23:10.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.053+0000 7ff68b7fe640 10 client.4353.objecter ms_handle_connect 0x7ff69006dd20 2024-03-29T11:23:10.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.057+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6940505a0 con 0x7ff6a4143d10 2024-03-29T11:23:10.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.057+0000 7ff68b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.057+0000 7ff68b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.418+0000 7ff6a920e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.418+0000 7ff6a920e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.418+0000 7ff6a920e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:10.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.418+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6a4074f50 con 0x7ff6a4143d10 2024-03-29T11:23:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.421+0000 7ff68b7fe640 1 -- 172.21.15.138:0/2195771465 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ff6940524d0 con 0x7ff6a4143d10 2024-03-29T11:23:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.421+0000 7ff68b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.421+0000 7ff68b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:10.399 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:10.399 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.424+0000 7ff6a920e640 10 librados: watch_flush enter 2024-03-29T11:23:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 10 librados: watch_flush exit 2024-03-29T11:23:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T11:23:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 10 client.4353.objecter successfully canceled tick 2024-03-29T11:23:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff69006dd20 legacy=0x7ff6900701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 10 monclient: shutdown 2024-03-29T11:23:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:10.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 >> v1:172.21.15.138:6789/0 conn(0x7ff6a4143d10 legacy=0x7ff6a41e7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 shutdown_connections 2024-03-29T11:23:10.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.425+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 >> 172.21.15.138:0/2195771465 conn(0x7ff6a4066a00 msgr2=0x7ff6a40b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:10.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.426+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 shutdown_connections 2024-03-29T11:23:10.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.426+0000 7ff6a920e640 1 -- 172.21.15.138:0/2195771465 wait complete. 2024-03-29T11:23:10.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:10.426+0000 7ff6a920e640 1 librados: shutdown 2024-03-29T11:23:10.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:17.230 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.405+0000 7fab1aee5640 10 monclient: get_monmap_and_config 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.405+0000 7fab1aee5640 10 monclient: build_initial_monmap 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.405+0000 7fab1aee5640 10 monclient: monmap: 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:17.405442+0000 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:17.405442+0000 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:17.383 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:17.384 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:17.384 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:17.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.406+0000 7fab1aee5640 1 Processor -- start 2024-03-29T11:23:17.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.406+0000 7fab1aee5640 1 -- start start 2024-03-29T11:23:17.384 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.406+0000 7fab1aee5640 10 monclient: init 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 10 monclient(hunting): picked mon.noname-c con 0x7fab0c006a70 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 10 monclient(hunting): picked mon.noname-b con 0x7fab0c02b100 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 10 monclient(hunting): picked mon.noname-a con 0x7fab0c029c30 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c02a270 con 0x7fab0c029c30 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c02c440 con 0x7fab0c006a70 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c0c9440 con 0x7fab0c02b100 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:17.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab1aee5640 10 monclient(hunting): authenticate will time out at 1018.715560s 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab196e2640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fab0c02b100 0x7fab0c0cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:58322/0 (socket says 172.21.15.138:58322) 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.407+0000 7fab196e2640 1 -- 172.21.15.138:0/2107544466 learned_addr learned my addr 172.21.15.138:0/2107544466 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.408+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344771420 0 0) 0x7fab0c0c9440 con 0x7fab0c02b100 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.408+0000 7fab18ee1640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.408+0000 7fab18ee1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.408+0000 7fab18ee1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.408+0000 7fab18ee1640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:23:17.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.408+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab00003520 con 0x7fab0c02b100 2024-03-29T11:23:17.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609912433 0 0) 0x7fab0c02a270 con 0x7fab0c029c30 2024-03-29T11:23:17.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c0c9440 con 0x7fab0c029c30 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1745276111 0 0) 0x7fab00003520 con 0x7fab0c02b100 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 >> v1:172.21.15.152:6789/0 conn(0x7fab0c006a70 legacy=0x7fab0c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 >> v1:172.21.15.138:6789/0 conn(0x7fab0c029c30 legacy=0x7fab0c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 10 monclient: _finish_hunting 0 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 1 monclient: found mon.noname-b 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.409+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c0c95d0 con 0x7fab0c02b100 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: _finish_auth 0 2024-03-29T11:23:17.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: _check_auth_tickets 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fab1404f730 con 0x7fab0c02b100 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: dump: 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:17.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab1aee5640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:23:17.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab1aee5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab140008d0 con 0x7fab0c02b100 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab1aee5640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab1aee5640 4 set_mon_vals no callback set 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 1 -- 172.21.15.138:0/2107544466 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fab140758b0 con 0x7fab0c02b100 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.410+0000 7fab18ee1640 10 monclient: dump: 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:17.391 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.411+0000 7fab1aee5640 10 monclient: shutdown 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.411+0000 7fab1aee5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.411+0000 7fab1aee5640 1 -- 172.21.15.138:0/2107544466 >> v1:172.21.15.138:6790/0 conn(0x7fab0c02b100 legacy=0x7fab0c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.411+0000 7fab1aee5640 1 -- 172.21.15.138:0/2107544466 shutdown_connections 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.411+0000 7fab1aee5640 1 -- 172.21.15.138:0/2107544466 >> 172.21.15.138:0/2107544466 conn(0x7fab0c0058f0 msgr2=0x7fab0c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.411+0000 7fab1aee5640 1 -- 172.21.15.138:0/2107544466 shutdown_connections 2024-03-29T11:23:17.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.411+0000 7fab1aee5640 1 -- 172.21.15.138:0/2107544466 wait complete. 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.412+0000 7fab1aee5640 10 monclient: build_initial_monmap 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.412+0000 7fab1aee5640 10 monclient: monmap: 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.413+0000 7fab1aee5640 1 librados: starting msgr at 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.413+0000 7fab1aee5640 1 librados: starting objecter 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.413+0000 7fab1aee5640 1 Processor -- start 2024-03-29T11:23:17.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.413+0000 7fab1aee5640 1 -- start start 2024-03-29T11:23:17.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.413+0000 7fab1aee5640 1 librados: setting wanted keys 2024-03-29T11:23:17.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.413+0000 7fab1aee5640 1 librados: calling monclient init 2024-03-29T11:23:17.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.413+0000 7fab1aee5640 10 monclient: init 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 10 monclient(hunting): picked mon.noname-b con 0x7fab0c006a70 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 10 monclient(hunting): picked mon.noname-c con 0x7fab0c029c30 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 10 monclient(hunting): picked mon.noname-a con 0x7fab0c0c2d00 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c0ccc60 con 0x7fab0c0c2d00 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c02d6b0 con 0x7fab0c006a70 2024-03-29T11:23:17.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c02e390 con 0x7fab0c029c30 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1aee5640 10 monclient(hunting): authenticate will time out at 1018.722234s 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1a6e4640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fab0c0c2d00 0x7fab0c160e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45832/0 (socket says 172.21.15.138:45832) 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab1a6e4640 1 -- 172.21.15.138:0/3042555393 learned_addr learned my addr 172.21.15.138:0/3042555393 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab0a7fc640 10 client.?.objecter ms_handle_connect 0x7fab0c0c2d00 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab0a7fc640 10 client.?.objecter ms_handle_connect 0x7fab0c006a70 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.414+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187501941 0 0) 0x7fab0c0ccc60 con 0x7fab0c0c2d00 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf8004eb0 con 0x7fab0c0c2d00 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 client.?.objecter ms_handle_connect 0x7fab0c029c30 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702352322 0 0) 0x7fab0c02d6b0 con 0x7fab0c006a70 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf80051d0 con 0x7fab0c006a70 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874870539 0 0) 0x7fab0c0c95d0 con 0x7fab0c029c30 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c02d6b0 con 0x7fab0c029c30 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375077422 0 0) 0x7faaf8004eb0 con 0x7fab0c0c2d00 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab0c0c95d0 con 0x7fab0c0c2d00 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fab040019d0 con 0x7fab0c0c2d00 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:17.399 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.415+0000 7fab0a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234163627 0 0) 0x7faaf80051d0 con 0x7fab0c006a70 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faaf8004eb0 con 0x7fab0c006a70 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fab10001830 con 0x7fab0c006a70 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:17.400 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1645764049 0 0) 0x7fab0c0c95d0 con 0x7fab0c0c2d00 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 >> v1:172.21.15.138:6790/0 conn(0x7fab0c029c30 legacy=0x7fab0c0c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 >> v1:172.21.15.152:6789/0 conn(0x7fab0c006a70 legacy=0x7fab0c02cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 monclient: found mon.a 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c02f390 con 0x7fab0c0c2d00 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:17.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 10 monclient: _renew_subs 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c02dd40 con 0x7fab0c0c2d00 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 10 monclient: _renew_subs 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c161560 con 0x7fab0c0c2d00 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.416+0000 7fab1aee5640 1 librados: init done 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.417+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab04001c90 con 0x7fab0c0c2d00 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.417+0000 7fab0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.417+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fab04003a60 con 0x7fab0c0c2d00 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.417+0000 7fab0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.417+0000 7fab0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.417+0000 7fab0a7fc640 10 monclient: dump: 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.417+0000 7fab0bfff640 4 set_mon_vals no callback set 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.418+0000 7fab1aee5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.418+0000 7fab1aee5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.418+0000 7fab1aee5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:17.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.418+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc005080 con 0x7fab0c0c2d00 2024-03-29T11:23:17.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.419+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fab04003bc0 con 0x7fab0c0c2d00 2024-03-29T11:23:17.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.419+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fab04084920 con 0x7fab0c0c2d00 2024-03-29T11:23:17.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.419+0000 7fab0a7fc640 10 client.4359.objecter ms_dispatch 0x7fab14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:17.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.419+0000 7fab0a7fc640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:17.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.419+0000 7fab0a7fc640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:17.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.419+0000 7fab0a7fc640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T11:23:17.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.419+0000 7fab19ee3640 10 monclient: get_auth_request con 0x7faaf806dfa0 auth_method 0 2024-03-29T11:23:17.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.420+0000 7fab0a7fc640 10 client.4359.objecter ms_handle_connect 0x7faaf806dfa0 2024-03-29T11:23:17.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.427+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab04050340 con 0x7fab0c0c2d00 2024-03-29T11:23:17.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.427+0000 7fab0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.427+0000 7fab0a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:17.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.836+0000 7fab1aee5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.836+0000 7fab1aee5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.836+0000 7fab1aee5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:17.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.836+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faafc005330 con 0x7fab0c0c2d00 2024-03-29T11:23:17.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.837+0000 7fab0a7fc640 1 -- 172.21.15.138:0/3042555393 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fab04050340 con 0x7fab0c0c2d00 2024-03-29T11:23:17.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.837+0000 7fab0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.837+0000 7fab0a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:17.815 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:17.815 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 10 librados: watch_flush enter 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 10 librados: watch_flush exit 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 10 client.4359.objecter successfully canceled tick 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 >> v1:172.21.15.152:6816/3052585572 conn(0x7faaf806dfa0 legacy=0x7faaf8070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 10 monclient: shutdown 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 >> v1:172.21.15.138:6789/0 conn(0x7fab0c0c2d00 legacy=0x7fab0c160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 shutdown_connections 2024-03-29T11:23:17.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 >> 172.21.15.138:0/3042555393 conn(0x7fab0c0058f0 msgr2=0x7fab0c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 shutdown_connections 2024-03-29T11:23:17.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 1 -- 172.21.15.138:0/3042555393 wait complete. 2024-03-29T11:23:17.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:17.841+0000 7fab1aee5640 1 librados: shutdown 2024-03-29T11:23:17.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:24.641 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.808+0000 7f4360515640 10 monclient: get_monmap_and_config 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.808+0000 7f4360515640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.808+0000 7f4360515640 10 monclient: monmap: 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:24.810021+0000 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:24.810021+0000 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.788 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:24.789 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:24.789 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:24.789 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:24.789 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.809+0000 7f4360515640 1 Processor -- start 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.809+0000 7f4360515640 1 -- start start 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.809+0000 7f4360515640 10 monclient: init 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 10 monclient(hunting): picked mon.noname-b con 0x7f43580b0770 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 10 monclient(hunting): picked mon.noname-c con 0x7f435814a700 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 10 monclient(hunting): picked mon.noname-a con 0x7f43580ac5a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43580aba70 con 0x7f43580ac5a0 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43580aecb0 con 0x7f435814a700 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43580af150 con 0x7f43580b0770 2024-03-29T11:23:24.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.810+0000 7f4360515640 10 monclient(hunting): authenticate will time out at 1026.119981s 2024-03-29T11:23:24.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435eab2640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f43580ac5a0 0x7f4358151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:54956/0 (socket says 172.21.15.138:54956) 2024-03-29T11:23:24.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435eab2640 1 -- 172.21.15.138:0/1373563643 learned_addr learned my addr 172.21.15.138:0/1373563643 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576190898 0 0) 0x7f43580aba70 con 0x7f43580ac5a0 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4344003520 con 0x7f43580ac5a0 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595624492 0 0) 0x7f43580af150 con 0x7f43580b0770 2024-03-29T11:23:24.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.811+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43580aba70 con 0x7f43580b0770 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246775476 0 0) 0x7f4344003520 con 0x7f43580ac5a0 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 >> v1:172.21.15.138:6790/0 conn(0x7f43580b0770 legacy=0x7f4358149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 >> v1:172.21.15.152:6789/0 conn(0x7f435814a700 legacy=0x7f43580abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 1 monclient: found mon.noname-a 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43580ac9d0 con 0x7f43580ac5a0 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f4360515640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f4360515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4350001a10 con 0x7f43580ac5a0 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: dump: 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:24.794 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f4360515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4350001bf0 con 0x7f43580ac5a0 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f435d2af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f4360515640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.812+0000 7f4360515640 4 set_mon_vals no callback set 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f435d2af640 1 -- 172.21.15.138:0/1373563643 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4350001a10 con 0x7f43580ac5a0 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f435d2af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f435d2af640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f435d2af640 10 monclient: dump: 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.795 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.796 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:24.796 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:24.796 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:24.796 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:24.796 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:24.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f4360515640 10 monclient: shutdown 2024-03-29T11:23:24.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f4360515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f4360515640 1 -- 172.21.15.138:0/1373563643 >> v1:172.21.15.138:6789/0 conn(0x7f43580ac5a0 legacy=0x7f4358151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f4360515640 1 -- 172.21.15.138:0/1373563643 shutdown_connections 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.813+0000 7f4360515640 1 -- 172.21.15.138:0/1373563643 >> 172.21.15.138:0/1373563643 conn(0x7f435806b660 msgr2=0x7f435806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.814+0000 7f4360515640 1 -- 172.21.15.138:0/1373563643 shutdown_connections 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.814+0000 7f4360515640 1 -- 172.21.15.138:0/1373563643 wait complete. 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 10 monclient: monmap: 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:24.800 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 1 librados: starting msgr at 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 1 librados: starting objecter 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 1 Processor -- start 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 1 -- start start 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 1 librados: setting wanted keys 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 1 librados: calling monclient init 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.815+0000 7f4360515640 10 monclient: init 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 10 monclient(hunting): picked mon.noname-c con 0x7f43580ac5a0 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f435e2b1640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f43580ac5a0 0x7f4358144fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:34608/0 (socket says 172.21.15.138:34608) 2024-03-29T11:23:24.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f435dab0640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f43580b0770 0x7f4358126850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:55234/0 (socket says 172.21.15.138:55234) 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f435e2b1640 1 -- 172.21.15.138:0/3201397033 learned_addr learned my addr 172.21.15.138:0/3201397033 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 10 monclient(hunting): picked mon.noname-a con 0x7f43580b0770 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 10 monclient(hunting): picked mon.noname-b con 0x7f435814a700 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4358128340 con 0x7f43580b0770 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4358128620 con 0x7f435814a700 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4358126330 con 0x7f43580ac5a0 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.816+0000 7f4360515640 10 monclient(hunting): authenticate will time out at 1026.125980s 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 client.?.objecter ms_handle_connect 0x7f43580b0770 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 client.?.objecter ms_handle_connect 0x7f43580ac5a0 2024-03-29T11:23:24.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091600613 0 0) 0x7f4358128340 con 0x7f43580b0770 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c004eb0 con 0x7f43580b0770 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307615388 0 0) 0x7f4358126330 con 0x7f43580ac5a0 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4358128340 con 0x7f43580ac5a0 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 661415964 0 0) 0x7f4358128340 con 0x7f43580ac5a0 2024-03-29T11:23:24.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4358126330 con 0x7f43580ac5a0 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4348001a30 con 0x7f43580ac5a0 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): dump: 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:24.804 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.817+0000 7f434effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3815248739 0 0) 0x7f433c004eb0 con 0x7f43580b0770 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4358128340 con 0x7f43580b0770 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4354003f20 con 0x7f43580b0770 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 10 monclient(hunting): dump: 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.805 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2241738203 0 0) 0x7f4358126330 con 0x7f43580ac5a0 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 >> v1:172.21.15.152:6789/0 conn(0x7f435814a700 legacy=0x7f4358126f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 >> v1:172.21.15.138:6789/0 conn(0x7f43580b0770 legacy=0x7f4358126850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 monclient: found mon.c 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4358125f50 con 0x7f43580ac5a0 2024-03-29T11:23:24.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f434effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 10 monclient: _renew_subs 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43581df960 con 0x7f43580ac5a0 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 10 monclient: _renew_subs 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43581df120 con 0x7f43580ac5a0 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.818+0000 7f4360515640 1 librados: init done 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4348001b90 con 0x7f43580ac5a0 2024-03-29T11:23:24.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f434effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4348003a40 con 0x7f43580ac5a0 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f434effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f434effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f434effd640 10 monclient: dump: 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:24.808 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:24.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f435caae640 4 set_mon_vals no callback set 2024-03-29T11:23:24.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.819+0000 7f4360515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.820+0000 7f4360515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.820+0000 7f4360515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:24.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.820+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4340005080 con 0x7f43580ac5a0 2024-03-29T11:23:24.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.820+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f4348003ba0 con 0x7f43580ac5a0 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.821+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f4348084b30 con 0x7f43580ac5a0 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.821+0000 7f435dab0640 10 monclient: get_auth_request con 0x7f433c06de80 auth_method 0 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.821+0000 7f434effd640 10 client.4367.objecter ms_dispatch 0x7f4358000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.821+0000 7f434effd640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.821+0000 7f434effd640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.821+0000 7f434effd640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.821+0000 7f434effd640 10 client.4367.objecter ms_handle_connect 0x7f433c06de80 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.830+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4348050240 con 0x7f43580ac5a0 2024-03-29T11:23:24.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.830+0000 7f434effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:24.830+0000 7f434effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:25.175 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.196+0000 7f4360515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.176 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.196+0000 7f4360515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.176 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.196+0000 7f4360515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:25.176 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.196+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4340005330 con 0x7f43580ac5a0 2024-03-29T11:23:25.176 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.197+0000 7f434effd640 1 -- 172.21.15.138:0/3201397033 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f4348052340 con 0x7f43580ac5a0 2024-03-29T11:23:25.176 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.197+0000 7f434effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.176 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.197+0000 7f434effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:25.176 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:25.177 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:25.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.200+0000 7f4360515640 10 librados: watch_flush enter 2024-03-29T11:23:25.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.200+0000 7f4360515640 10 librados: watch_flush exit 2024-03-29T11:23:25.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.200+0000 7f4360515640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T11:23:25.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.200+0000 7f4360515640 10 client.4367.objecter successfully canceled tick 2024-03-29T11:23:25.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.200+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 >> v1:172.21.15.152:6816/3052585572 conn(0x7f433c06de80 legacy=0x7f433c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 10 monclient: shutdown 2024-03-29T11:23:25.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:25.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 >> v1:172.21.15.138:6790/0 conn(0x7f43580ac5a0 legacy=0x7f4358144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 shutdown_connections 2024-03-29T11:23:25.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 >> 172.21.15.138:0/3201397033 conn(0x7f435806b660 msgr2=0x7f4358071650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:25.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 shutdown_connections 2024-03-29T11:23:25.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 1 -- 172.21.15.138:0/3201397033 wait complete. 2024-03-29T11:23:25.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:25.201+0000 7f4360515640 1 librados: shutdown 2024-03-29T11:23:25.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:32.003 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.151+0000 7fef11c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.151+0000 7fef11c4e640 10 monclient: build_initial_monmap 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.151+0000 7fef11c4e640 10 monclient: monmap: 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:32.152795+0000 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:32.152795+0000 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:32.131 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:32.132 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:32.132 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:32.132 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:32.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.152+0000 7fef11c4e640 1 Processor -- start 2024-03-29T11:23:32.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.152+0000 7fef11c4e640 1 -- start start 2024-03-29T11:23:32.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.152+0000 7fef11c4e640 10 monclient: init 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fef0c1422b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fef0c142690 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fef0c0b17d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c142c80 con 0x7fef0c142690 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c0aae40 con 0x7fef0c1422b0 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c13fbd0 con 0x7fef0c0b17d0 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef11c4e640 10 monclient(hunting): authenticate will time out at 1033.462820s 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef0bfff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fef0c0b17d0 0x7fef0c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:48436/0 (socket says 172.21.15.138:48436) 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.153+0000 7fef0bfff640 1 -- 172.21.15.138:0/2846446315 learned_addr learned my addr 172.21.15.138:0/2846446315 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:32.135 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246355741 0 0) 0x7fef0c13fbd0 con 0x7fef0c0b17d0 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef4003520 con 0x7fef0c0b17d0 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25458726 0 0) 0x7feef4003520 con 0x7fef0c0b17d0 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 >> v1:172.21.15.152:6789/0 conn(0x7fef0c1422b0 legacy=0x7fef0c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:32.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.154+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 >> v1:172.21.15.138:6789/0 conn(0x7fef0c142690 legacy=0x7fef0c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:32.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c151940 con 0x7fef0c0b17d0 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7feefc001670 con 0x7fef0c0b17d0 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef0a7fc640 10 monclient: dump: 2024-03-29T11:23:32.138 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:32.139 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:32.139 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.139 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.139 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.139 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:32.139 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:32.139 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef11c4e640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.157+0000 7fef11c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feefc0018d0 con 0x7fef0c0b17d0 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef11c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef11c4e640 4 set_mon_vals no callback set 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef0a7fc640 1 -- 172.21.15.138:0/2846446315 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7feefc001ca0 con 0x7fef0c0b17d0 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:32.140 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.158+0000 7fef0a7fc640 10 monclient: dump: 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:32.141 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.159+0000 7fef11c4e640 10 monclient: shutdown 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.159+0000 7fef11c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.159+0000 7fef11c4e640 1 -- 172.21.15.138:0/2846446315 >> v1:172.21.15.138:6790/0 conn(0x7fef0c0b17d0 legacy=0x7fef0c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.159+0000 7fef11c4e640 1 -- 172.21.15.138:0/2846446315 shutdown_connections 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.159+0000 7fef11c4e640 1 -- 172.21.15.138:0/2846446315 >> 172.21.15.138:0/2846446315 conn(0x7fef0c066a00 msgr2=0x7fef0c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.159+0000 7fef11c4e640 1 -- 172.21.15.138:0/2846446315 shutdown_connections 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.159+0000 7fef11c4e640 1 -- 172.21.15.138:0/2846446315 wait complete. 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.160+0000 7fef11c4e640 10 monclient: build_initial_monmap 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.160+0000 7fef11c4e640 10 monclient: monmap: 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.161+0000 7fef11c4e640 1 librados: starting msgr at 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.161+0000 7fef11c4e640 1 librados: starting objecter 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.161+0000 7fef11c4e640 1 Processor -- start 2024-03-29T11:23:32.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.161+0000 7fef11c4e640 1 -- start start 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.161+0000 7fef11c4e640 1 librados: setting wanted keys 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.161+0000 7fef11c4e640 1 librados: calling monclient init 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.161+0000 7fef11c4e640 10 monclient: init 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.162+0000 7fef11c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.162+0000 7fef11c4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fef0c0b17d0 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fef0c1422b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fef0c142690 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c0ac770 con 0x7fef0c0b17d0 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c0abd90 con 0x7fef0c1422b0 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c0ac9a0 con 0x7fef0c142690 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef11c4e640 10 monclient(hunting): authenticate will time out at 1033.472793s 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef0bfff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fef0c142690 0x7fef0c12fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:49148/0 (socket says 172.21.15.138:49148) 2024-03-29T11:23:32.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7fef0bfff640 1 -- 172.21.15.138:0/3103606505 learned_addr learned my addr 172.21.15.138:0/3103606505 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7feef3fff640 10 client.?.objecter ms_handle_connect 0x7fef0c142690 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.163+0000 7feef3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186426728 0 0) 0x7fef0c0ac9a0 con 0x7fef0c142690 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef8004eb0 con 0x7fef0c142690 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2725821426 0 0) 0x7feef8004eb0 con 0x7fef0c142690 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef0c0ac9a0 con 0x7fef0c142690 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7feefc001be0 con 0x7fef0c142690 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:32.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): dump: 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1166725349 0 0) 0x7fef0c0ac9a0 con 0x7fef0c142690 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 >> v1:172.21.15.152:6789/0 conn(0x7fef0c1422b0 legacy=0x7fef0c0b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.164+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 >> v1:172.21.15.138:6789/0 conn(0x7fef0c0b17d0 legacy=0x7fef0c132340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 1 monclient: found mon.c 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c0acc80 con 0x7fef0c142690 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.165+0000 7feef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 10 monclient: _renew_subs 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0c0acf60 con 0x7fef0c142690 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feefc004bf0 con 0x7fef0c142690 2024-03-29T11:23:32.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7feef3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7feefc004f00 con 0x7fef0c142690 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7feef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7feef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7feef3fff640 10 monclient: dump: 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:32.156 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 10 monclient: _renew_subs 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0c0b1240 con 0x7fef0c142690 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef09ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.166+0000 7fef11c4e640 1 librados: init done 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.169+0000 7fef11c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.169+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7feefc005060 con 0x7fef0c142690 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.169+0000 7fef11c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.169+0000 7fef11c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.169+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0c0abf20 con 0x7fef0c142690 2024-03-29T11:23:32.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.170+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7feefc084cf0 con 0x7fef0c142690 2024-03-29T11:23:32.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.170+0000 7feef3fff640 10 client.4373.objecter ms_dispatch 0x7fef0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:32.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.170+0000 7feef3fff640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:32.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.170+0000 7feef3fff640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:32.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.170+0000 7feef3fff640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:23:32.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.170+0000 7fef0b7fe640 10 monclient: get_auth_request con 0x7feef806dbc0 auth_method 0 2024-03-29T11:23:32.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.170+0000 7feef3fff640 10 client.4373.objecter ms_handle_connect 0x7feef806dbc0 2024-03-29T11:23:32.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.177+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef0c0abf20 con 0x7fef0c142690 2024-03-29T11:23:32.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.177+0000 7feef3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.177+0000 7feef3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:32.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.473+0000 7fef11c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.473+0000 7fef11c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.473+0000 7fef11c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:32.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.473+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef0c0abf20 con 0x7fef0c142690 2024-03-29T11:23:32.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.473+0000 7feef3fff640 1 -- 172.21.15.138:0/3103606505 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fef0c0abf20 con 0x7fef0c142690 2024-03-29T11:23:32.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.473+0000 7feef3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.473+0000 7feef3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:32.454 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:32.454 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 10 librados: watch_flush enter 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 10 librados: watch_flush exit 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 >> v1:172.21.15.152:6816/3052585572 conn(0x7feef806dbc0 legacy=0x7feef8070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 10 monclient: shutdown 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:32.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 >> v1:172.21.15.138:6790/0 conn(0x7fef0c142690 legacy=0x7fef0c12fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 shutdown_connections 2024-03-29T11:23:32.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.477+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 >> 172.21.15.138:0/3103606505 conn(0x7fef0c066a00 msgr2=0x7fef0c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:32.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.478+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 shutdown_connections 2024-03-29T11:23:32.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.478+0000 7fef11c4e640 1 -- 172.21.15.138:0/3103606505 wait complete. 2024-03-29T11:23:32.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:32.478+0000 7fef11c4e640 1 librados: shutdown 2024-03-29T11:23:32.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:39.281 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:39.427 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.447+0000 7f5ae5e96640 10 monclient: get_monmap_and_config 2024-03-29T11:23:39.427 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.447+0000 7f5ae5e96640 10 monclient: build_initial_monmap 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.447+0000 7f5ae5e96640 10 monclient: monmap: 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:39.448915+0000 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:39.448915+0000 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:39.428 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:39.430 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.450+0000 7f5ae5e96640 1 Processor -- start 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.451+0000 7f5ae5e96640 1 -- start start 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.451+0000 7f5ae5e96640 10 monclient: init 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae00b0770 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae014a700 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae00ac5a0 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae00aba70 con 0x7f5ae00b0770 2024-03-29T11:23:39.432 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae00aecb0 con 0x7f5ae014a700 2024-03-29T11:23:39.433 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae00af150 con 0x7f5ae00ac5a0 2024-03-29T11:23:39.433 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:39.433 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.452+0000 7f5ae5e96640 10 monclient(hunting): authenticate will time out at 1040.761849s 2024-03-29T11:23:39.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.456+0000 7f5adeffd640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f5ae014a700 0x7f5ae00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:47829/0 (socket says 172.21.15.138:47829) 2024-03-29T11:23:39.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.456+0000 7f5adeffd640 1 -- 172.21.15.138:0/203379629 learned_addr learned my addr 172.21.15.138:0/203379629 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:39.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401175678 0 0) 0x7f5ae00aecb0 con 0x7f5ae014a700 2024-03-29T11:23:39.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac8003520 con 0x7f5ae014a700 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458472042 0 0) 0x7f5ac8003520 con 0x7f5ae014a700 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 >> v1:172.21.15.138:6790/0 conn(0x7f5ae00ac5a0 legacy=0x7f5ae0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 >> v1:172.21.15.138:6789/0 conn(0x7f5ae00b0770 legacy=0x7f5ae0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 monclient: found mon.noname-c 2024-03-29T11:23:39.437 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae00ac9d0 con 0x7f5ae014a700 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5ad00016b0 con 0x7f5ae014a700 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.457+0000 7f5ade7fc640 10 monclient: dump: 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.438 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad0001940 con 0x7f5ae014a700 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ade7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ade7fc640 1 -- 172.21.15.138:0/203379629 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5ad00016b0 con 0x7f5ae014a700 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ade7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ade7fc640 10 monclient: dump: 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:39.439 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.440 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.440 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.440 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:39.440 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:39.440 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:39.440 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:39.440 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ae5e96640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ae5e96640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.458+0000 7f5ae5e96640 4 set_mon_vals no callback set 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.459+0000 7f5ae5e96640 10 monclient: shutdown 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.459+0000 7f5ae5e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.459+0000 7f5ae5e96640 1 -- 172.21.15.138:0/203379629 >> v1:172.21.15.152:6789/0 conn(0x7f5ae014a700 legacy=0x7f5ae00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.460+0000 7f5ae5e96640 1 -- 172.21.15.138:0/203379629 shutdown_connections 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.460+0000 7f5ae5e96640 1 -- 172.21.15.138:0/203379629 >> 172.21.15.138:0/203379629 conn(0x7f5ae006b660 msgr2=0x7f5ae006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.460+0000 7f5ae5e96640 1 -- 172.21.15.138:0/203379629 shutdown_connections 2024-03-29T11:23:39.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.460+0000 7f5ae5e96640 1 -- 172.21.15.138:0/203379629 wait complete. 2024-03-29T11:23:39.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.461+0000 7f5ae5e96640 10 monclient: build_initial_monmap 2024-03-29T11:23:39.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.461+0000 7f5ae5e96640 10 monclient: monmap: 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.461+0000 7f5ae5e96640 1 librados: starting msgr at 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.461+0000 7f5ae5e96640 1 librados: starting objecter 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.461+0000 7f5ae5e96640 1 Processor -- start 2024-03-29T11:23:39.446 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 1 -- start start 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 1 librados: setting wanted keys 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 1 librados: calling monclient init 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient: init 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae00ac5a0 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae00b0770 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae0144b10 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae01df8d0 con 0x7f5ae0144b10 2024-03-29T11:23:39.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae01dfb70 con 0x7f5ae00ac5a0 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae0145a20 con 0x7f5ae00b0770 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.462+0000 7f5ae5e96640 10 monclient(hunting): authenticate will time out at 1040.771979s 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.463+0000 7f5adeffd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f5ae00b0770 0x7f5ae0144400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:57716/0 (socket says 172.21.15.138:57716) 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.463+0000 7f5adeffd640 1 -- 172.21.15.138:0/3360424058 learned_addr learned my addr 172.21.15.138:0/3360424058 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.463+0000 7f5ac7fff640 10 client.?.objecter ms_handle_connect 0x7f5ae00b0770 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.463+0000 7f5ac7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.464+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683030663 0 0) 0x7f5ae0145a20 con 0x7f5ae00b0770 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.464+0000 7f5ac7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.464+0000 7f5ac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.464+0000 7f5ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.464+0000 7f5ac7fff640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.464+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5acc004eb0 con 0x7f5ae00b0770 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3580343479 0 0) 0x7f5acc004eb0 con 0x7f5ae00b0770 2024-03-29T11:23:39.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae0145a20 con 0x7f5ae00b0770 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5ad00016b0 con 0x7f5ae00b0770 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:39.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:39.450 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1545547300 0 0) 0x7f5ae0145a20 con 0x7f5ae00b0770 2024-03-29T11:23:39.450 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.465+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 >> v1:172.21.15.152:6789/0 conn(0x7f5ae00ac5a0 legacy=0x7f5ae01df1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 >> v1:172.21.15.138:6789/0 conn(0x7f5ae0144b10 legacy=0x7f5ae0126210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 1 monclient: found mon.c 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae0144f80 con 0x7f5ae00b0770 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:39.451 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 10 monclient: _renew_subs 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 1 -- 172.21.15.138:0/3360424058 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae0145170 con 0x7f5ae00b0770 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 10 monclient: _renew_subs 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 1 -- 172.21.15.138:0/3360424058 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae0126b00 con 0x7f5ae00b0770 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.468+0000 7f5ae5e96640 1 librados: init done 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac7fff640 10 client.4379.objecter ms_handle_connect 0x7f5ae0144b10 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac7fff640 10 client.4379.objecter resend_mon_ops 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac0005080 con 0x7f5ae00b0770 2024-03-29T11:23:39.452 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad0001c90 con 0x7f5ae00b0770 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5ac7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.470+0000 7f5addffb640 4 set_mon_vals no callback set 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.471+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5ad0005050 con 0x7f5ae00b0770 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.471+0000 7f5ac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.471+0000 7f5ac7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.471+0000 7f5ac7fff640 10 monclient: dump: 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:39.453 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:39.454 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.472+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f5ad0005270 con 0x7f5ae00b0770 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.472+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f5ad0084bf0 con 0x7f5ae00b0770 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.472+0000 7f5ac7fff640 10 client.4379.objecter ms_dispatch 0x7f5ae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.472+0000 7f5ac7fff640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.472+0000 7f5ac7fff640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.472+0000 7f5ac7fff640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.473+0000 7f5adf7fe640 10 monclient: get_auth_request con 0x7f5acc06db10 auth_method 0 2024-03-29T11:23:39.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.473+0000 7f5ac7fff640 10 client.4379.objecter ms_handle_connect 0x7f5acc06db10 2024-03-29T11:23:39.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.479+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ad00504d0 con 0x7f5ae00b0770 2024-03-29T11:23:39.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.479+0000 7f5ac7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.479+0000 7f5ac7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:39.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.798+0000 7f5ac5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.798+0000 7f5ac5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.798+0000 7f5ac5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:39.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.798+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ac0005330 con 0x7f5ae00b0770 2024-03-29T11:23:39.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.799+0000 7f5ac7fff640 1 -- 172.21.15.138:0/3360424058 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f5ad0052400 con 0x7f5ae00b0770 2024-03-29T11:23:39.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.799+0000 7f5ac7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.799+0000 7f5ac7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:39.779 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:39.780 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:39.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.802+0000 7f5ac5ffb640 10 librados: watch_flush enter 2024-03-29T11:23:39.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.802+0000 7f5ac5ffb640 10 librados: watch_flush exit 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.802+0000 7f5ac5ffb640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.802+0000 7f5ac5ffb640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.803+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 >> v1:172.21.15.152:6816/3052585572 conn(0x7f5acc06db10 legacy=0x7f5acc06ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.803+0000 7f5ac5ffb640 10 monclient: shutdown 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.803+0000 7f5ac5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.803+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 >> v1:172.21.15.138:6790/0 conn(0x7f5ae00b0770 legacy=0x7f5ae0144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.803+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 shutdown_connections 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.803+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 >> 172.21.15.138:0/3360424058 conn(0x7f5ae006b660 msgr2=0x7f5ae006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:39.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.803+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 shutdown_connections 2024-03-29T11:23:39.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.804+0000 7f5ac5ffb640 1 -- 172.21.15.138:0/3360424058 wait complete. 2024-03-29T11:23:39.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:39.804+0000 7f5ac5ffb640 1 librados: shutdown 2024-03-29T11:23:39.801 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:46.607 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:46.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.776+0000 7f3000c31640 10 monclient: get_monmap_and_config 2024-03-29T11:23:46.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.776+0000 7f3000c31640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.776+0000 7f3000c31640 10 monclient: monmap: 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:46.777351+0000 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:46.777351+0000 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:46.757 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:46.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.777+0000 7f3000c31640 1 Processor -- start 2024-03-29T11:23:46.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.777+0000 7f3000c31640 1 -- start start 2024-03-29T11:23:46.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.777+0000 7f3000c31640 10 monclient: init 2024-03-29T11:23:46.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f3000c31640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f3000c31640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:46.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f3000c31640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff4006800 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:46.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f2ffb7fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f2ff4006800 0x7f2ff40cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:43564/0 (socket says 172.21.15.138:43564) 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f2ffb7fe640 1 -- 172.21.15.138:0/3908685588 learned_addr learned my addr 172.21.15.138:0/3908685588 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f3000c31640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff4006be0 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f3000c31640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff4031b60 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff4029370 con 0x7f2ff4006be0 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.778+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40295e0 con 0x7f2ff4031b60 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.779+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff4031360 con 0x7f2ff4006800 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.779+0000 7f3000c31640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.760 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.779+0000 7f3000c31640 10 monclient(hunting): authenticate will time out at 1048.088287s 2024-03-29T11:23:46.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.781+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435928082 0 0) 0x7f2ff4031360 con 0x7f2ff4006800 2024-03-29T11:23:46.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.781+0000 7f2ffa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.781+0000 7f2ffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.781+0000 7f2ffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.781+0000 7f2ffa7fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:23:46.762 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.781+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fec003520 con 0x7f2ff4006800 2024-03-29T11:23:46.762 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290744678 0 0) 0x7f2fec003520 con 0x7f2ff4006800 2024-03-29T11:23:46.762 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 >> v1:172.21.15.152:6789/0 conn(0x7f2ff4031b60 legacy=0x7f2ff40d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:46.762 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 >> v1:172.21.15.138:6789/0 conn(0x7f2ff4006be0 legacy=0x7f2ff40c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:46.762 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.762 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4031640 con 0x7f2ff4006800 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2ffc04f730 con 0x7f2ff4006800 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.782+0000 7f2ffa7fc640 10 monclient: dump: 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.763 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.783+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ffc075f10 con 0x7f2ff4006800 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.783+0000 7f2ffa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.783+0000 7f2ffa7fc640 1 -- 172.21.15.138:0/3908685588 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2ffc0758b0 con 0x7f2ff4006800 2024-03-29T11:23:46.764 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.783+0000 7f2ffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.783+0000 7f2ffa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.783+0000 7f2ffa7fc640 10 monclient: dump: 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:46.765 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:46.766 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.785+0000 7f3000c31640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:23:46.766 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.785+0000 7f3000c31640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:46.766 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.785+0000 7f3000c31640 4 set_mon_vals no callback set 2024-03-29T11:23:46.766 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.786+0000 7f3000c31640 10 monclient: shutdown 2024-03-29T11:23:46.766 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.786+0000 7f3000c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:46.767 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.786+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 >> v1:172.21.15.138:6790/0 conn(0x7f2ff4006800 legacy=0x7f2ff40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.767 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.786+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 shutdown_connections 2024-03-29T11:23:46.767 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.786+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 >> 172.21.15.138:0/3908685588 conn(0x7f2ff40058f0 msgr2=0x7f2ff4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:46.767 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.786+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 shutdown_connections 2024-03-29T11:23:46.767 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.786+0000 7f3000c31640 1 -- 172.21.15.138:0/3908685588 wait complete. 2024-03-29T11:23:46.769 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.787+0000 7f3000c31640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.787+0000 7f3000c31640 10 monclient: monmap: 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.787+0000 7f3000c31640 1 librados: starting msgr at 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.787+0000 7f3000c31640 1 librados: starting objecter 2024-03-29T11:23:46.770 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.787+0000 7f3000c31640 1 Processor -- start 2024-03-29T11:23:46.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.788+0000 7f3000c31640 1 -- start start 2024-03-29T11:23:46.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.788+0000 7f3000c31640 1 librados: setting wanted keys 2024-03-29T11:23:46.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.788+0000 7f3000c31640 1 librados: calling monclient init 2024-03-29T11:23:46.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.788+0000 7f3000c31640 10 monclient: init 2024-03-29T11:23:46.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.789+0000 7f3000c31640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.789+0000 7f3000c31640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:46.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.790+0000 7f3000c31640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff4006800 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f3000c31640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff4031b60 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f3000c31640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff415c490 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f3000c31640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff415cb00 con 0x7f2ff4031b60 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f3000c31640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40c5cb0 con 0x7f2ff4006800 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f3000c31640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40bf0a0 con 0x7f2ff415c490 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f3000c31640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f3000c31640 10 monclient(hunting): authenticate will time out at 1048.101509s 2024-03-29T11:23:46.776 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f2ffb7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f2ff4006800 0x7f2ff40c6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:44623/0 (socket says 172.21.15.138:44623) 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.792+0000 7f2ffb7fe640 1 -- 172.21.15.138:0/1374414635 learned_addr learned my addr 172.21.15.138:0/1374414635 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 client.?.objecter ms_handle_connect 0x7f2ff4006800 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 client.?.objecter ms_handle_connect 0x7f2ff415c490 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231142328 0 0) 0x7f2ff40bf0a0 con 0x7f2ff415c490 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe0004eb0 con 0x7f2ff415c490 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733406283 0 0) 0x7f2ff40c5cb0 con 0x7f2ff4006800 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.777 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.793+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff40bf0a0 con 0x7f2ff4006800 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1615398792 0 0) 0x7f2fe0004eb0 con 0x7f2ff415c490 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ff40c5cb0 con 0x7f2ff415c490 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2fe8001750 con 0x7f2ff415c490 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.778 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2279728275 0 0) 0x7f2ff40bf0a0 con 0x7f2ff4006800 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fe0004eb0 con 0x7f2ff4006800 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 489459415 0 0) 0x7f2ff40c5cb0 con 0x7f2ff415c490 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 >> v1:172.21.15.152:6789/0 conn(0x7f2ff4006800 legacy=0x7f2ff40c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 >> v1:172.21.15.138:6789/0 conn(0x7f2ff4031b60 legacy=0x7f2ff40c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.794+0000 7f2fe7fff640 1 monclient: found mon.c 2024-03-29T11:23:46.779 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff40c8760 con 0x7f2ff415c490 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fe80018b0 con 0x7f2ff415c490 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2ff9ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2fe8003790 con 0x7f2ff415c490 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.795+0000 7f2fe7fff640 10 monclient: dump: 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.780 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:46.781 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.781 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:46.781 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:46.781 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:46.781 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:46.781 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:46.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 10 monclient: _renew_subs 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff40c7c30 con 0x7f2ff415c490 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 10 monclient: _renew_subs 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff40bf940 con 0x7f2ff415c490 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.797+0000 7f3000c31640 1 librados: init done 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.801+0000 7f3000c31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.801+0000 7f3000c31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.801+0000 7f3000c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.801+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff4004cd0 con 0x7f2ff415c490 2024-03-29T11:23:46.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.802+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f2fe8019760 con 0x7f2ff415c490 2024-03-29T11:23:46.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.802+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f2fe80836a0 con 0x7f2ff415c490 2024-03-29T11:23:46.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.802+0000 7f2fe7fff640 10 client.4385.objecter ms_dispatch 0x7f2ffc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:46.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.802+0000 7f2fe7fff640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:46.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.802+0000 7f2fe7fff640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:46.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.803+0000 7f2fe7fff640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:23:46.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.804+0000 7f2ffb7fe640 10 monclient: get_auth_request con 0x7f2fe006dde0 auth_method 0 2024-03-29T11:23:46.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.806+0000 7f2fe7fff640 10 client.4385.objecter ms_handle_connect 0x7f2fe006dde0 2024-03-29T11:23:46.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.810+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fe804ef80 con 0x7f2ff415c490 2024-03-29T11:23:46.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.810+0000 7f2fe7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:46.810+0000 7f2fe7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:47.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.133+0000 7f3000c31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:47.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.133+0000 7f3000c31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:47.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.133+0000 7f3000c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:23:47.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.133+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ff40c7a00 con 0x7f2ff415c490 2024-03-29T11:23:47.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.134+0000 7f2fe7fff640 1 -- 172.21.15.138:0/1374414635 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f2fe8050eb0 con 0x7f2ff415c490 2024-03-29T11:23:47.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.134+0000 7f2fe7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:47.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.134+0000 7f2fe7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:47.117 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:47.117 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 10 librados: watch_flush enter 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 10 librados: watch_flush exit 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 >> v1:172.21.15.152:6816/3052585572 conn(0x7f2fe006dde0 legacy=0x7f2fe00702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 10 monclient: shutdown 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.140+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 >> v1:172.21.15.138:6790/0 conn(0x7f2ff415c490 legacy=0x7f2ff40c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.141+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 shutdown_connections 2024-03-29T11:23:47.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.141+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 >> 172.21.15.138:0/1374414635 conn(0x7f2ff40058f0 msgr2=0x7f2ff402a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:47.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.142+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 shutdown_connections 2024-03-29T11:23:47.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.142+0000 7f3000c31640 1 -- 172.21.15.138:0/1374414635 wait complete. 2024-03-29T11:23:47.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:47.142+0000 7f3000c31640 1 librados: shutdown 2024-03-29T11:23:47.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:53.947 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:54.095 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.114+0000 7f7dc61cf640 10 monclient: get_monmap_and_config 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.114+0000 7f7dc61cf640 10 monclient: build_initial_monmap 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.114+0000 7f7dc61cf640 10 monclient: monmap: 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:23:54.116204+0000 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:23:54.116204+0000 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:23:54.096 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:54.097 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.116+0000 7f7dc61cf640 1 Processor -- start 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 1 -- start start 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient: init 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc00b0770 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc014a700 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc00ac5a0 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc00aba70 con 0x7f7dc00b0770 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc00aecb0 con 0x7f7dc014a700 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc00af150 con 0x7f7dc00ac5a0 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:54.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.117+0000 7f7dc61cf640 10 monclient(hunting): authenticate will time out at 1055.427043s 2024-03-29T11:23:54.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.122+0000 7f7dbeffd640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f7dc014a700 0x7f7dc00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:37569/0 (socket says 172.21.15.138:37569) 2024-03-29T11:23:54.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.123+0000 7f7dbeffd640 1 -- 172.21.15.138:0/3118829571 learned_addr learned my addr 172.21.15.138:0/3118829571 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:54.104 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.123+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189487681 0 0) 0x7f7dc00aecb0 con 0x7f7dc014a700 2024-03-29T11:23:54.104 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.123+0000 7f7dbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:54.104 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.123+0000 7f7dbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:54.104 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.123+0000 7f7dbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:54.104 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.123+0000 7f7dbe7fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.124+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da8003520 con 0x7f7dc014a700 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.124+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649538832 0 0) 0x7f7da8003520 con 0x7f7dc014a700 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.124+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 >> v1:172.21.15.138:6790/0 conn(0x7f7dc00ac5a0 legacy=0x7f7dc0151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 >> v1:172.21.15.138:6789/0 conn(0x7f7dc00b0770 legacy=0x7f7dc0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 1 monclient: found mon.noname-c 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc00ac9d0 con 0x7f7dc014a700 2024-03-29T11:23:54.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:54.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:54.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:54.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dc61cf640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:23:54.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.126+0000 7f7dc61cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7db8001a30 con 0x7f7dc014a700 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 10 monclient: dump: 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:54.111 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7db8001d00 con 0x7f7dc014a700 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 1 -- 172.21.15.138:0/3118829571 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7db80020d0 con 0x7f7dc014a700 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.129+0000 7f7dbe7fc640 10 monclient: dump: 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:54.112 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:54.113 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:54.113 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:54.113 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:54.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.131+0000 7f7dc61cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:54.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.131+0000 7f7dc61cf640 4 set_mon_vals no callback set 2024-03-29T11:23:54.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.132+0000 7f7dc61cf640 10 monclient: shutdown 2024-03-29T11:23:54.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.132+0000 7f7dc61cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:54.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.132+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3118829571 >> v1:172.21.15.152:6789/0 conn(0x7f7dc014a700 legacy=0x7f7dc00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.132+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3118829571 shutdown_connections 2024-03-29T11:23:54.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.132+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3118829571 >> 172.21.15.138:0/3118829571 conn(0x7f7dc006b660 msgr2=0x7f7dc006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:54.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.133+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3118829571 shutdown_connections 2024-03-29T11:23:54.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.133+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3118829571 wait complete. 2024-03-29T11:23:54.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.134+0000 7f7dc61cf640 10 monclient: build_initial_monmap 2024-03-29T11:23:54.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.134+0000 7f7dc61cf640 10 monclient: monmap: 2024-03-29T11:23:54.115 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:23:54.115 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:54.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.135+0000 7f7dc61cf640 1 librados: starting msgr at 2024-03-29T11:23:54.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.135+0000 7f7dc61cf640 1 librados: starting objecter 2024-03-29T11:23:54.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.135+0000 7f7dc61cf640 1 Processor -- start 2024-03-29T11:23:54.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.135+0000 7f7dc61cf640 1 -- start start 2024-03-29T11:23:54.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.135+0000 7f7dc61cf640 1 librados: setting wanted keys 2024-03-29T11:23:54.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.135+0000 7f7dc61cf640 1 librados: calling monclient init 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.135+0000 7f7dc61cf640 10 monclient: init 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc00ac5a0 addr v1:172.21.15.138:6790/0 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc00b0770 addr v1:172.21.15.152:6789/0 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc0145070 addr v1:172.21.15.138:6789/0 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc01dfa90 con 0x7f7dc0145070 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc01288d0 con 0x7f7dc00b0770 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc0145b10 con 0x7f7dc00ac5a0 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:54.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.136+0000 7f7dc61cf640 10 monclient(hunting): authenticate will time out at 1055.445611s 2024-03-29T11:23:54.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.141+0000 7f7dbffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f7dc0145070 0x7f7dc0126270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:55051/0 (socket says 172.21.15.138:55051) 2024-03-29T11:23:54.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.141+0000 7f7dbffff640 1 -- 172.21.15.138:0/3291603724 learned_addr learned my addr 172.21.15.138:0/3291603724 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:23:54.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.141+0000 7f7da7fff640 10 client.?.objecter ms_handle_connect 0x7f7dc0145070 2024-03-29T11:23:54.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.141+0000 7f7da7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:54.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.143+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415991472 0 0) 0x7f7dc01dfa90 con 0x7f7dc0145070 2024-03-29T11:23:54.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.143+0000 7f7da7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:54.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.143+0000 7f7da7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:54.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.143+0000 7f7da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:54.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.143+0000 7f7da7fff640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:23:54.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.143+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac004eb0 con 0x7f7dc0145070 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1453012103 0 0) 0x7f7dac004eb0 con 0x7f7dc0145070 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dc01dfa90 con 0x7f7dc0145070 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7db0004000 con 0x7f7dc0145070 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:54.126 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2480559324 0 0) 0x7f7dc01dfa90 con 0x7f7dc0145070 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 >> v1:172.21.15.138:6790/0 conn(0x7f7dc00ac5a0 legacy=0x7f7dc01df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 >> v1:172.21.15.152:6789/0 conn(0x7f7dc00b0770 legacy=0x7f7dc0144380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 monclient: found mon.a 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:54.127 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:54.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc0144a90 con 0x7f7dc0145070 2024-03-29T11:23:54.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:54.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:54.128 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.145+0000 7f7da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:54.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:23:54.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 10 monclient: _renew_subs 2024-03-29T11:23:54.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:54.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc0144d40 con 0x7f7dc0145070 2024-03-29T11:23:54.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:54.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 10 monclient: _renew_subs 2024-03-29T11:23:54.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:54.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc0126b60 con 0x7f7dc0145070 2024-03-29T11:23:54.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.146+0000 7f7dc61cf640 1 librados: init done 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.148+0000 7f7dc61cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.148+0000 7f7dc61cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.148+0000 7f7dc61cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.148+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc00b1aa0 con 0x7f7dc0145070 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.151+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7db0000ed0 con 0x7f7dc0145070 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.151+0000 7f7da7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.151+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7db0004000 con 0x7f7dc0145070 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.151+0000 7f7da7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.151+0000 7f7da7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.151+0000 7f7da7fff640 10 monclient: dump: 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:23:54.133 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.151+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f7db0001180 con 0x7f7dc0145070 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.152+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f7db00848a0 con 0x7f7dc0145070 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.152+0000 7f7da7fff640 10 client.4374.objecter ms_dispatch 0x7f7dc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.152+0000 7f7da7fff640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.152+0000 7f7da7fff640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.152+0000 7f7da7fff640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.152+0000 7f7dbd7fa640 4 set_mon_vals no callback set 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.153+0000 7f7dbf7fe640 10 monclient: get_auth_request con 0x7f7dac06dee0 auth_method 0 2024-03-29T11:23:54.134 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.154+0000 7f7da7fff640 10 client.4374.objecter ms_handle_connect 0x7f7dac06dee0 2024-03-29T11:23:54.139 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.159+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7db0050200 con 0x7f7dc0145070 2024-03-29T11:23:54.139 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.159+0000 7f7da7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:54.139 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.159+0000 7f7da7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:54.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.477+0000 7f7dc61cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.477+0000 7f7dc61cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.477+0000 7f7dc61cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:23:54.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.477+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dc0126e60 con 0x7f7dc0145070 2024-03-29T11:23:54.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.478+0000 7f7da7fff640 1 -- 172.21.15.138:0/3291603724 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f7db0052130 con 0x7f7dc0145070 2024-03-29T11:23:54.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.478+0000 7f7da7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.478+0000 7f7da7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:54.459 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:23:54.459 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:54.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.482+0000 7f7dc61cf640 10 librados: watch_flush enter 2024-03-29T11:23:54.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.482+0000 7f7dc61cf640 10 librados: watch_flush exit 2024-03-29T11:23:54.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.482+0000 7f7dc61cf640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:23:54.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.482+0000 7f7dc61cf640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:23:54.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.484+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 >> v1:172.21.15.152:6816/3052585572 conn(0x7f7dac06dee0 legacy=0x7f7dac0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.484+0000 7f7dc61cf640 10 monclient: shutdown 2024-03-29T11:23:54.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.484+0000 7f7dc61cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:54.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.484+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 >> v1:172.21.15.138:6789/0 conn(0x7f7dc0145070 legacy=0x7f7dc0126270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.484+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 shutdown_connections 2024-03-29T11:23:54.465 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.484+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 >> 172.21.15.138:0/3291603724 conn(0x7f7dc006b660 msgr2=0x7f7dc00b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:54.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.485+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 shutdown_connections 2024-03-29T11:23:54.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.485+0000 7f7dc61cf640 1 -- 172.21.15.138:0/3291603724 wait complete. 2024-03-29T11:23:54.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:23:54.485+0000 7f7dc61cf640 1 librados: shutdown 2024-03-29T11:23:54.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:01.290 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.465+0000 7faa5c8c9640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.465+0000 7faa5c8c9640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.465+0000 7faa5c8c9640 10 monclient: monmap: 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:01.465430+0000 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:01.465430+0000 2024-03-29T11:24:01.445 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.446 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:01.446 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:01.446 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:01.446 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:01.446 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:01.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.467+0000 7faa5c8c9640 1 Processor -- start 2024-03-29T11:24:01.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.467+0000 7faa5c8c9640 1 -- start start 2024-03-29T11:24:01.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.467+0000 7faa5c8c9640 10 monclient: init 2024-03-29T11:24:01.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.468+0000 7faa5c8c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.468+0000 7faa5c8c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:01.454 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.471+0000 7faa5c8c9640 10 monclient(hunting): picked mon.noname-a con 0x7faa50006a70 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:01.454 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.472+0000 7faa5c8c9640 10 monclient(hunting): picked mon.noname-b con 0x7faa5002b100 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:01.454 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.472+0000 7faa5c8c9640 10 monclient(hunting): picked mon.noname-c con 0x7faa50029c30 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:01.454 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.473+0000 7faa5c8c9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5002a270 con 0x7faa50006a70 2024-03-29T11:24:01.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.473+0000 7faa5c8c9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5002c440 con 0x7faa50029c30 2024-03-29T11:24:01.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.473+0000 7faa5c8c9640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500c9440 con 0x7faa5002b100 2024-03-29T11:24:01.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.473+0000 7faa5c8c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.473+0000 7faa5c8c9640 10 monclient(hunting): authenticate will time out at 1062.781150s 2024-03-29T11:24:01.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.473+0000 7faa577fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7faa50006a70 0x7faa500c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:53459/0 (socket says 172.21.15.138:53459) 2024-03-29T11:24:01.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.473+0000 7faa577fe640 1 -- 172.21.15.138:0/2462576499 learned_addr learned my addr 172.21.15.138:0/2462576499 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:01.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300616110 0 0) 0x7faa5002a270 con 0x7faa50006a70 2024-03-29T11:24:01.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:24:01.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa48003520 con 0x7faa50006a70 2024-03-29T11:24:01.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3395772693 0 0) 0x7faa48003520 con 0x7faa50006a70 2024-03-29T11:24:01.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.474+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 >> v1:172.21.15.138:6790/0 conn(0x7faa5002b100 legacy=0x7faa500cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:01.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 >> v1:172.21.15.152:6789/0 conn(0x7faa50029c30 legacy=0x7faa500d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:01.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 1 monclient: found mon.noname-a 2024-03-29T11:24:01.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:24:01.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500c95d0 con 0x7faa50006a70 2024-03-29T11:24:01.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.475+0000 7faa567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa44001690 con 0x7faa50006a70 2024-03-29T11:24:01.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:01.464 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 10 monclient: dump: 2024-03-29T11:24:01.465 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:01.465 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:01.465 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.465 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.465 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.465 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:01.466 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:01.466 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:01.466 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:01.466 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:01.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa5c8c9640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:24:01.466 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa5c8c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.467 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa44001860 con 0x7faa50006a70 2024-03-29T11:24:01.467 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.467 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 1 -- 172.21.15.138:0/2462576499 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa44001cb0 con 0x7faa50006a70 2024-03-29T11:24:01.467 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.467 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:01.467 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa567fc640 10 monclient: dump: 2024-03-29T11:24:01.468 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:01.468 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:01.468 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.468 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.468 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.468 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:01.468 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:01.469 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:01.469 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:01.469 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:01.469 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa5c8c9640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.469 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.476+0000 7faa5c8c9640 4 set_mon_vals no callback set 2024-03-29T11:24:01.469 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.477+0000 7faa5c8c9640 10 monclient: shutdown 2024-03-29T11:24:01.470 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.477+0000 7faa5c8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.470 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.477+0000 7faa5c8c9640 1 -- 172.21.15.138:0/2462576499 >> v1:172.21.15.138:6789/0 conn(0x7faa50006a70 legacy=0x7faa500c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.470 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.477+0000 7faa5c8c9640 1 -- 172.21.15.138:0/2462576499 shutdown_connections 2024-03-29T11:24:01.470 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.477+0000 7faa5c8c9640 1 -- 172.21.15.138:0/2462576499 >> 172.21.15.138:0/2462576499 conn(0x7faa500058f0 msgr2=0x7faa50005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.470 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.478+0000 7faa5c8c9640 1 -- 172.21.15.138:0/2462576499 shutdown_connections 2024-03-29T11:24:01.470 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.480+0000 7faa5c8c9640 1 -- 172.21.15.138:0/2462576499 wait complete. 2024-03-29T11:24:01.471 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.481+0000 7faa5c8c9640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.471 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.481+0000 7faa5c8c9640 10 monclient: monmap: 2024-03-29T11:24:01.471 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:01.471 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.471 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:01.471 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:01.472 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.472 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:01.472 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:01.472 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:01.472 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:01.472 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:01.473 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.481+0000 7faa5c8c9640 1 librados: starting msgr at 2024-03-29T11:24:01.473 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.481+0000 7faa5c8c9640 1 librados: starting objecter 2024-03-29T11:24:01.473 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.481+0000 7faa5c8c9640 1 Processor -- start 2024-03-29T11:24:01.473 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.482+0000 7faa5c8c9640 1 -- start start 2024-03-29T11:24:01.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.482+0000 7faa5c8c9640 1 librados: setting wanted keys 2024-03-29T11:24:01.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.482+0000 7faa5c8c9640 1 librados: calling monclient init 2024-03-29T11:24:01.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.482+0000 7faa5c8c9640 10 monclient: init 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 10 monclient(hunting): picked mon.noname-a con 0x7faa50006a70 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 10 monclient(hunting): picked mon.noname-c con 0x7faa50029c30 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 10 monclient(hunting): picked mon.noname-b con 0x7faa500cae50 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500cca30 con 0x7faa50006a70 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500cb370 con 0x7faa500cae50 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500c9980 con 0x7faa50029c30 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.483+0000 7faa5c8c9640 10 monclient(hunting): authenticate will time out at 1062.791539s 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.489+0000 7faa577fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7faa50006a70 0x7faa500ca5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:54099/0 (socket says 172.21.15.138:54099) 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.489+0000 7faa577fe640 1 -- 172.21.15.138:0/3484790635 learned_addr learned my addr 172.21.15.138:0/3484790635 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.489+0000 7faa43fff640 10 client.?.objecter ms_handle_connect 0x7faa50006a70 2024-03-29T11:24:01.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.489+0000 7faa43fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.489+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344349262 0 0) 0x7faa500cca30 con 0x7faa50006a70 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.489+0000 7faa43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.490+0000 7faa43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.490+0000 7faa43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.490+0000 7faa43fff640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.490+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa3c004eb0 con 0x7faa50006a70 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.493+0000 7faa43fff640 10 client.?.objecter ms_handle_connect 0x7faa500cae50 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.493+0000 7faa43fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.493+0000 7faa43fff640 10 client.?.objecter ms_handle_connect 0x7faa50029c30 2024-03-29T11:24:01.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.493+0000 7faa43fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.493+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3839156198 0 0) 0x7faa3c004eb0 con 0x7faa50006a70 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.493+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa500cca30 con 0x7faa50006a70 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa44001b70 con 0x7faa50006a70 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): dump: 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:01.486 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216881267 0 0) 0x7faa500c9980 con 0x7faa50029c30 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa3c004eb0 con 0x7faa50029c30 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990087700 0 0) 0x7faa500cb370 con 0x7faa500cae50 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa500c9980 con 0x7faa500cae50 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3033572280 0 0) 0x7faa500cca30 con 0x7faa50006a70 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 >> v1:172.21.15.138:6790/0 conn(0x7faa50029c30 legacy=0x7faa500bf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 >> v1:172.21.15.152:6789/0 conn(0x7faa500cae50 legacy=0x7faa50033530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 monclient: found mon.a 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:01.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500bff70 con 0x7faa50006a70 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 10 monclient: _renew_subs 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa500c01c0 con 0x7faa50006a70 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 10 monclient: _renew_subs 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.494+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa50033e20 con 0x7faa50006a70 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa5c8c9640 1 librados: init done 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa44004c60 con 0x7faa50006a70 2024-03-29T11:24:01.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faa440050b0 con 0x7faa50006a70 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa43fff640 10 monclient: dump: 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:01.490 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.495+0000 7faa55ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.498+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7faa44005210 con 0x7faa50006a70 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.499+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7faa44084c80 con 0x7faa50006a70 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.499+0000 7faa56ffd640 10 monclient: get_auth_request con 0x7faa3c06dd50 auth_method 0 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.499+0000 7faa43fff640 10 client.4380.objecter ms_dispatch 0x7faa58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.499+0000 7faa43fff640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.499+0000 7faa43fff640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.500+0000 7faa43fff640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.500+0000 7faa43fff640 10 client.4380.objecter ms_handle_connect 0x7faa3c06dd50 2024-03-29T11:24:01.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.500+0000 7faa5c8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.500+0000 7faa5c8c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.500+0000 7faa5c8c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:01.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.500+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa38005080 con 0x7faa50006a70 2024-03-29T11:24:01.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.509+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa44084e60 con 0x7faa50006a70 2024-03-29T11:24:01.492 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.509+0000 7faa43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.509+0000 7faa43fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.819 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:01.819 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:01.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.835+0000 7faa5c8c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.835+0000 7faa5c8c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.835+0000 7faa5c8c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:01.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.835+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa38005330 con 0x7faa50006a70 2024-03-29T11:24:01.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.838+0000 7faa43fff640 1 -- 172.21.15.138:0/3484790635 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7faa44050390 con 0x7faa50006a70 2024-03-29T11:24:01.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.838+0000 7faa43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.838+0000 7faa43fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 10 librados: watch_flush enter 2024-03-29T11:24:01.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 10 librados: watch_flush exit 2024-03-29T11:24:01.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:24:01.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 >> v1:172.21.15.152:6816/3052585572 conn(0x7faa3c06dd50 legacy=0x7faa3c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 10 monclient: shutdown 2024-03-29T11:24:01.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 >> v1:172.21.15.138:6789/0 conn(0x7faa50006a70 legacy=0x7faa500ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 shutdown_connections 2024-03-29T11:24:01.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.842+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 >> 172.21.15.138:0/3484790635 conn(0x7faa500058f0 msgr2=0x7faa500372e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.825 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.845+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 shutdown_connections 2024-03-29T11:24:01.825 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.846+0000 7faa5c8c9640 1 -- 172.21.15.138:0/3484790635 wait complete. 2024-03-29T11:24:01.826 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:01.846+0000 7faa5c8c9640 1 librados: shutdown 2024-03-29T11:24:01.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:08.652 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.822+0000 7f07c0732640 10 monclient: get_monmap_and_config 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.822+0000 7f07c0732640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.822+0000 7f07c0732640 10 monclient: monmap: 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:08.822896+0000 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:08.822896+0000 2024-03-29T11:24:08.804 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.823+0000 7f07c0732640 1 Processor -- start 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.823+0000 7f07c0732640 1 -- start start 2024-03-29T11:24:08.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.823+0000 7f07c0732640 10 monclient: init 2024-03-29T11:24:08.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:08.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 10 monclient(hunting): picked mon.noname-b con 0x7f07b80a9750 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:08.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 10 monclient(hunting): picked mon.noname-c con 0x7f07b8149ac0 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:08.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 10 monclient(hunting): picked mon.noname-a con 0x7f07b814fe50 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:08.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b80aa4a0 con 0x7f07b814fe50 2024-03-29T11:24:08.809 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b814a680 con 0x7f07b8149ac0 2024-03-29T11:24:08.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b80af290 con 0x7f07b80a9750 2024-03-29T11:24:08.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.824+0000 7f07c0732640 10 monclient(hunting): authenticate will time out at 1070.132837s 2024-03-29T11:24:08.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.828+0000 7f07beccf640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f07b814fe50 0x7f07b8152240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48951/0 (socket says 172.21.15.138:48951) 2024-03-29T11:24:08.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.828+0000 7f07beccf640 1 -- 172.21.15.138:0/720677200 learned_addr learned my addr 172.21.15.138:0/720677200 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:08.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.829+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619343582 0 0) 0x7f07b80af290 con 0x7f07b80a9750 2024-03-29T11:24:08.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.829+0000 7f07bd4cc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.829+0000 7f07bd4cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.829+0000 7f07bd4cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.829+0000 7f07bd4cc640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:24:08.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.829+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a4003520 con 0x7f07b80a9750 2024-03-29T11:24:08.811 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410723839 0 0) 0x7f07a4003520 con 0x7f07b80a9750 2024-03-29T11:24:08.813 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 >> v1:172.21.15.152:6789/0 conn(0x7f07b8149ac0 legacy=0x7f07b8149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:08.813 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 >> v1:172.21.15.138:6789/0 conn(0x7f07b814fe50 legacy=0x7f07b8152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.813 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.813 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 1 monclient: found mon.noname-b 2024-03-29T11:24:08.813 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b80afb40 con 0x7f07b80a9750 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07c0732640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07c0732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f07b0001a10 con 0x7f07b80a9750 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07bd4cc640 10 monclient: dump: 2024-03-29T11:24:08.814 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:08.815 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.830+0000 7f07c0732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:08.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.833+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07b0001ca0 con 0x7f07b80a9750 2024-03-29T11:24:08.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.833+0000 7f07bd4cc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.833+0000 7f07c0732640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:08.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.833+0000 7f07c0732640 4 set_mon_vals no callback set 2024-03-29T11:24:08.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.834+0000 7f07bd4cc640 1 -- 172.21.15.138:0/720677200 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f07b0002030 con 0x7f07b80a9750 2024-03-29T11:24:08.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.834+0000 7f07bd4cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.816 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.834+0000 7f07bd4cc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.834+0000 7f07bd4cc640 10 monclient: dump: 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.834+0000 7f07c0732640 10 monclient: shutdown 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.834+0000 7f07c0732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:08.817 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.834+0000 7f07c0732640 1 -- 172.21.15.138:0/720677200 >> v1:172.21.15.138:6790/0 conn(0x7f07b80a9750 legacy=0x7f07b81436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.835+0000 7f07c0732640 1 -- 172.21.15.138:0/720677200 shutdown_connections 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.835+0000 7f07c0732640 1 -- 172.21.15.138:0/720677200 >> 172.21.15.138:0/720677200 conn(0x7f07b80b35e0 msgr2=0x7f07b80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.835+0000 7f07c0732640 1 -- 172.21.15.138:0/720677200 shutdown_connections 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.835+0000 7f07c0732640 1 -- 172.21.15.138:0/720677200 wait complete. 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.836+0000 7f07c0732640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.836+0000 7f07c0732640 10 monclient: monmap: 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:08.818 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:08.819 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:08.819 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:08.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.836+0000 7f07c0732640 1 librados: starting msgr at 2024-03-29T11:24:08.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.836+0000 7f07c0732640 1 librados: starting objecter 2024-03-29T11:24:08.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.836+0000 7f07c0732640 1 Processor -- start 2024-03-29T11:24:08.819 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 1 -- start start 2024-03-29T11:24:08.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 1 librados: setting wanted keys 2024-03-29T11:24:08.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 1 librados: calling monclient init 2024-03-29T11:24:08.820 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient: init 2024-03-29T11:24:08.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:08.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient(hunting): picked mon.noname-a con 0x7f07b80a9750 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:08.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient(hunting): picked mon.noname-c con 0x7f07b8149ac0 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:08.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient(hunting): picked mon.noname-b con 0x7f07b814fe50 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:08.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b80ad0d0 con 0x7f07b80a9750 2024-03-29T11:24:08.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b8149620 con 0x7f07b814fe50 2024-03-29T11:24:08.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b81497b0 con 0x7f07b8149ac0 2024-03-29T11:24:08.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.837+0000 7f07c0732640 10 monclient(hunting): authenticate will time out at 1070.145951s 2024-03-29T11:24:08.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.842+0000 7f07beccf640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f07b814fe50 0x7f07b81e5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:46971/0 (socket says 172.21.15.138:46971) 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.842+0000 7f07beccf640 1 -- 172.21.15.138:0/2896013083 learned_addr learned my addr 172.21.15.138:0/2896013083 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.842+0000 7f07aeffd640 10 client.?.objecter ms_handle_connect 0x7f07b814fe50 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.842+0000 7f07aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.843+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566877933 0 0) 0x7f07b8149620 con 0x7f07b814fe50 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.843+0000 7f07aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.843+0000 7f07aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.843+0000 7f07aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.823 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.843+0000 7f07aeffd640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.843+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f079c004eb0 con 0x7f07b814fe50 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.844+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 733520119 0 0) 0x7f079c004eb0 con 0x7f07b814fe50 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.844+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07b8149620 con 0x7f07b814fe50 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.844+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f07b40018b0 con 0x7f07b814fe50 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.844+0000 7f07aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.844+0000 7f07aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.844+0000 7f07aeffd640 10 monclient(hunting): dump: 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:08.824 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:08.825 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.844+0000 7f07aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:08.828 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1273845773 0 0) 0x7f07b8149620 con 0x7f07b814fe50 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 >> v1:172.21.15.138:6790/0 conn(0x7f07b8149ac0 legacy=0x7f07b8148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 >> v1:172.21.15.138:6789/0 conn(0x7f07b80a9750 legacy=0x7f07b80ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 1 monclient: found mon.b 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b8148140 con 0x7f07b814fe50 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 10 monclient: _renew_subs 2024-03-29T11:24:08.829 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b81478c0 con 0x7f07b814fe50 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 10 monclient: _renew_subs 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b81e7800 con 0x7f07b814fe50 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07c0732640 1 librados: init done 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07b4001a10 con 0x7f07b814fe50 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f07b4004a80 con 0x7f07b814fe50 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.848+0000 7f07aeffd640 10 monclient: dump: 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:08.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.849+0000 7f07affff640 4 set_mon_vals no callback set 2024-03-29T11:24:08.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.850+0000 7f07c0732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.850+0000 7f07c0732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.850+0000 7f07c0732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:08.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.850+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a0005080 con 0x7f07b814fe50 2024-03-29T11:24:08.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.852+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f07b4004cc0 con 0x7f07b814fe50 2024-03-29T11:24:08.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.853+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f07b4084be0 con 0x7f07b814fe50 2024-03-29T11:24:08.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.853+0000 7f07aeffd640 10 client.4342.objecter ms_dispatch 0x7f07b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:08.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.853+0000 7f07aeffd640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:08.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.853+0000 7f07aeffd640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:08.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.853+0000 7f07aeffd640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T11:24:08.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.854+0000 7f07be4ce640 10 monclient: get_auth_request con 0x7f079c06dc00 auth_method 0 2024-03-29T11:24:08.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.855+0000 7f07aeffd640 10 client.4342.objecter ms_handle_connect 0x7f079c06dc00 2024-03-29T11:24:08.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.862+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07b40504c0 con 0x7f07b814fe50 2024-03-29T11:24:08.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.862+0000 7f07aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:08.862+0000 7f07aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:09.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.192+0000 7f07c0732640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:09.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.192+0000 7f07c0732640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:09.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.192+0000 7f07c0732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:09.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.192+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07a0005330 con 0x7f07b814fe50 2024-03-29T11:24:09.174 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.193+0000 7f07aeffd640 1 -- 172.21.15.138:0/2896013083 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f07b40523f0 con 0x7f07b814fe50 2024-03-29T11:24:09.174 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.194+0000 7f07aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:09.174 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.194+0000 7f07aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:09.175 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:09.175 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:09.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.198+0000 7f07c0732640 10 librados: watch_flush enter 2024-03-29T11:24:09.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.199+0000 7f07c0732640 10 librados: watch_flush exit 2024-03-29T11:24:09.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.199+0000 7f07c0732640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T11:24:09.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.199+0000 7f07c0732640 10 client.4342.objecter successfully canceled tick 2024-03-29T11:24:09.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.199+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 >> v1:172.21.15.152:6816/3052585572 conn(0x7f079c06dc00 legacy=0x7f079c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.199+0000 7f07c0732640 10 monclient: shutdown 2024-03-29T11:24:09.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.199+0000 7f07c0732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:09.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.199+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 >> v1:172.21.15.152:6789/0 conn(0x7f07b814fe50 legacy=0x7f07b81e5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.200+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 shutdown_connections 2024-03-29T11:24:09.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.200+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 >> 172.21.15.138:0/2896013083 conn(0x7f07b80b35e0 msgr2=0x7f07b81434c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:09.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.200+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 shutdown_connections 2024-03-29T11:24:09.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.200+0000 7f07c0732640 1 -- 172.21.15.138:0/2896013083 wait complete. 2024-03-29T11:24:09.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:09.200+0000 7f07c0732640 1 librados: shutdown 2024-03-29T11:24:09.199 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:16.006 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.181+0000 7fabfd063640 10 monclient: get_monmap_and_config 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.181+0000 7fabfd063640 10 monclient: build_initial_monmap 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.181+0000 7fabfd063640 10 monclient: monmap: 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:16.182903+0000 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:16.182903+0000 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:16.164 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:16.164 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:16.164 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:16.164 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:16.164 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:16.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.184+0000 7fabfd063640 1 Processor -- start 2024-03-29T11:24:16.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.184+0000 7fabfd063640 1 -- start start 2024-03-29T11:24:16.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.184+0000 7fabfd063640 10 monclient: init 2024-03-29T11:24:16.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.185+0000 7fabfd063640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:16.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.185+0000 7fabfd063640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 10 monclient(hunting): picked mon.noname-c con 0x7fabf80b2790 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 10 monclient(hunting): picked mon.noname-b con 0x7fabf80b37d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 10 monclient(hunting): picked mon.noname-a con 0x7fabf8146940 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80b2b70 con 0x7fabf8146940 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf806eec0 con 0x7fabf80b2790 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf806f1a0 con 0x7fabf80b37d0 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:16.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabfd063640 10 monclient(hunting): authenticate will time out at 1077.498642s 2024-03-29T11:24:16.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.189+0000 7fabf6d9d640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fabf80b2790 0x7fabf8143360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:59399/0 (socket says 172.21.15.138:59399) 2024-03-29T11:24:16.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.190+0000 7fabf6d9d640 1 -- 172.21.15.138:0/107355656 learned_addr learned my addr 172.21.15.138:0/107355656 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:16.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914567264 0 0) 0x7fabf806eec0 con 0x7fabf80b2790 2024-03-29T11:24:16.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:16.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:16.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:16.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0003520 con 0x7fabf80b2790 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 359140764 0 0) 0x7fabe0003520 con 0x7fabf80b2790 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 >> v1:172.21.15.138:6790/0 conn(0x7fabf80b37d0 legacy=0x7fabf8146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 >> v1:172.21.15.138:6789/0 conn(0x7fabf8146940 legacy=0x7fabf8151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80b2e20 con 0x7fabf80b2790 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:24:16.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fabf0001a40 con 0x7fabf80b2790 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.195+0000 7fabf5d9b640 10 monclient: dump: 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:16.182 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabf0001c10 con 0x7fabf80b2790 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabf5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabf5d9b640 1 -- 172.21.15.138:0/107355656 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fabf0002020 con 0x7fabf80b2790 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabf5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabf5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabf5d9b640 10 monclient: dump: 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:16.183 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabfd063640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabfd063640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.196+0000 7fabfd063640 4 set_mon_vals no callback set 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.197+0000 7fabfd063640 10 monclient: shutdown 2024-03-29T11:24:16.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.197+0000 7fabfd063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:16.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.197+0000 7fabfd063640 1 -- 172.21.15.138:0/107355656 >> v1:172.21.15.152:6789/0 conn(0x7fabf80b2790 legacy=0x7fabf8143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.199+0000 7fabfd063640 1 -- 172.21.15.138:0/107355656 shutdown_connections 2024-03-29T11:24:16.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.199+0000 7fabfd063640 1 -- 172.21.15.138:0/107355656 >> 172.21.15.138:0/107355656 conn(0x7fabf806d180 msgr2=0x7fabf8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:16.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.199+0000 7fabfd063640 1 -- 172.21.15.138:0/107355656 shutdown_connections 2024-03-29T11:24:16.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.202+0000 7fabfd063640 1 -- 172.21.15.138:0/107355656 wait complete. 2024-03-29T11:24:16.190 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.203+0000 7fabfd063640 10 monclient: build_initial_monmap 2024-03-29T11:24:16.190 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.203+0000 7fabfd063640 10 monclient: monmap: 2024-03-29T11:24:16.190 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:16.190 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.203+0000 7fabfd063640 1 librados: starting msgr at 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.203+0000 7fabfd063640 1 librados: starting objecter 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.204+0000 7fabfd063640 1 Processor -- start 2024-03-29T11:24:16.191 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.204+0000 7fabfd063640 1 -- start start 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.204+0000 7fabfd063640 1 librados: setting wanted keys 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.204+0000 7fabfd063640 1 librados: calling monclient init 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.204+0000 7fabfd063640 10 monclient: init 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.205+0000 7fabfd063640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.205+0000 7fabfd063640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 10 monclient(hunting): picked mon.noname-b con 0x7fabf80b2790 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 10 monclient(hunting): picked mon.noname-a con 0x7fabf80b37d0 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 10 monclient(hunting): picked mon.noname-c con 0x7fabf8146940 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf81483e0 con 0x7fabf80b37d0 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf81e87a0 con 0x7fabf80b2790 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf8147640 con 0x7fabf8146940 2024-03-29T11:24:16.192 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.207+0000 7fabfd063640 10 monclient(hunting): authenticate will time out at 1077.516374s 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.208+0000 7fabf6d9d640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fabf80b2790 0x7fabf8148d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:59533/0 (socket says 172.21.15.138:59533) 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.208+0000 7fabf6d9d640 1 -- 172.21.15.138:0/1090292951 learned_addr learned my addr 172.21.15.138:0/1090292951 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 client.?.objecter ms_handle_connect 0x7fabf80b2790 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596699316 0 0) 0x7fabf81e87a0 con 0x7fabf80b2790 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:24:16.193 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe4004eb0 con 0x7fabf80b2790 2024-03-29T11:24:16.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 233028753 0 0) 0x7fabe4004eb0 con 0x7fabf80b2790 2024-03-29T11:24:16.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabf81e87a0 con 0x7fabf80b2790 2024-03-29T11:24:16.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fabf0001f20 con 0x7fabf80b2790 2024-03-29T11:24:16.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:16.194 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.209+0000 7fabdf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.210+0000 7fabdf7fe640 10 client.?.objecter ms_handle_connect 0x7fabf8146940 2024-03-29T11:24:16.195 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.210+0000 7fabdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:16.196 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.210+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3393280035 0 0) 0x7fabf81e87a0 con 0x7fabf80b2790 2024-03-29T11:24:16.196 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.210+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 >> v1:172.21.15.138:6790/0 conn(0x7fabf8146940 legacy=0x7fabf81e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.196 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.210+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 >> v1:172.21.15.138:6789/0 conn(0x7fabf80b37d0 legacy=0x7fabf81e7980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:16.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:16.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 1 monclient: found mon.b 2024-03-29T11:24:16.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:16.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:16.198 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf8148690 con 0x7fabf80b2790 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.213+0000 7fabdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.216+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabf0004fa0 con 0x7fabf80b2790 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.216+0000 7fabdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.216+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fabf00052b0 con 0x7fabf80b2790 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.216+0000 7fabdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.216+0000 7fabdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:16.200 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.216+0000 7fabdf7fe640 10 monclient: dump: 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 10 monclient: _renew_subs 2024-03-29T11:24:16.201 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:16.202 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf8148970 con 0x7fabf80b2790 2024-03-29T11:24:16.202 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.216+0000 7fabf4d99640 4 set_mon_vals no callback set 2024-03-29T11:24:16.202 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:16.202 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 10 monclient: _renew_subs 2024-03-29T11:24:16.202 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:16.202 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf814a340 con 0x7fabf80b2790 2024-03-29T11:24:16.202 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.217+0000 7fabfd063640 1 librados: init done 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.219+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fabf001b280 con 0x7fabf80b2790 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.220+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fabf0084ff0 con 0x7fabf80b2790 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.220+0000 7fabdf7fe640 10 client.4348.objecter ms_dispatch 0x7fabf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.220+0000 7fabdf7fe640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.220+0000 7fabdf7fe640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.220+0000 7fabdf7fe640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.220+0000 7fabf659c640 10 monclient: get_auth_request con 0x7fabe406dcb0 auth_method 0 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.220+0000 7fabdf7fe640 10 client.4348.objecter ms_handle_connect 0x7fabe406dcb0 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.221+0000 7fabfd063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.221+0000 7fabfd063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:16.204 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.221+0000 7fabfd063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:16.205 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.221+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf814a690 con 0x7fabf80b2790 2024-03-29T11:24:16.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.227+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabf00508d0 con 0x7fabf80b2790 2024-03-29T11:24:16.209 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.227+0000 7fabdf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:16.209 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.227+0000 7fabdf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:16.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.576+0000 7fabfd063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.576+0000 7fabfd063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.576+0000 7fabfd063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:16.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.576+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabf81409c0 con 0x7fabf80b2790 2024-03-29T11:24:16.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.577+0000 7fabdf7fe640 1 -- 172.21.15.138:0/1090292951 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fabf0052800 con 0x7fabf80b2790 2024-03-29T11:24:16.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.577+0000 7fabdf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.577+0000 7fabdf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:16.561 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:16.561 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:16.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 10 librados: watch_flush enter 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 10 librados: watch_flush exit 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 >> v1:172.21.15.152:6816/3052585572 conn(0x7fabe406dcb0 legacy=0x7fabe4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 10 monclient: shutdown 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.582+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 >> v1:172.21.15.152:6789/0 conn(0x7fabf80b2790 legacy=0x7fabf8148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.583+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 shutdown_connections 2024-03-29T11:24:16.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.583+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 >> 172.21.15.138:0/1090292951 conn(0x7fabf806d180 msgr2=0x7fabf81428f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:16.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.586+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 shutdown_connections 2024-03-29T11:24:16.572 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.589+0000 7fabfd063640 1 -- 172.21.15.138:0/1090292951 wait complete. 2024-03-29T11:24:16.572 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:16.590+0000 7fabfd063640 1 librados: shutdown 2024-03-29T11:24:16.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:23.396 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:23.552 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.570+0000 7f932ee7b640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.570+0000 7f932ee7b640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.570+0000 7f932ee7b640 10 monclient: monmap: 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:23.572215+0000 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:23.572215+0000 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:23.553 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:23.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.572+0000 7f932ee7b640 1 Processor -- start 2024-03-29T11:24:23.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.572+0000 7f932ee7b640 1 -- start start 2024-03-29T11:24:23.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.572+0000 7f932ee7b640 10 monclient: init 2024-03-29T11:24:23.555 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.573+0000 7f932ee7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.555 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.573+0000 7f932ee7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:23.555 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.573+0000 7f932ee7b640 10 monclient(hunting): picked mon.noname-b con 0x7f93280b4920 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:23.555 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.574+0000 7f932ee7b640 10 monclient(hunting): picked mon.noname-c con 0x7f93280b4200 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:23.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.574+0000 7f932ee7b640 10 monclient(hunting): picked mon.noname-a con 0x7f932813f9a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:23.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.574+0000 7f932ee7b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93280b0930 con 0x7f932813f9a0 2024-03-29T11:24:23.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.574+0000 7f932ee7b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93280af460 con 0x7f93280b4200 2024-03-29T11:24:23.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.574+0000 7f932ee7b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93280aeda0 con 0x7f93280b4920 2024-03-29T11:24:23.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.574+0000 7f932ee7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.574+0000 7f932ee7b640 10 monclient(hunting): authenticate will time out at 1084.883593s 2024-03-29T11:24:23.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.575+0000 7f932e67a640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f932813f9a0 0x7f932814d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:51882/0 (socket says 172.21.15.138:51882) 2024-03-29T11:24:23.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.575+0000 7f932e67a640 1 -- 172.21.15.138:0/3797073173 learned_addr learned my addr 172.21.15.138:0/3797073173 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:23.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.575+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792819267 0 0) 0x7f93280b0930 con 0x7f932813f9a0 2024-03-29T11:24:23.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.575+0000 7f932ce77640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.575+0000 7f932ce77640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.575+0000 7f932ce77640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.575+0000 7f932ce77640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:24:23.558 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.576+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9314003520 con 0x7f932813f9a0 2024-03-29T11:24:23.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.576+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187532074 0 0) 0x7f93280aeda0 con 0x7f93280b4920 2024-03-29T11:24:23.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.576+0000 7f932ce77640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.576+0000 7f932ce77640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:24:23.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93280b0930 con 0x7f93280b4920 2024-03-29T11:24:23.561 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 845609152 0 0) 0x7f9314003520 con 0x7f932813f9a0 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 >> v1:172.21.15.138:6790/0 conn(0x7f93280b4920 legacy=0x7f932813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 >> v1:172.21.15.152:6789/0 conn(0x7f93280b4200 legacy=0x7f93280b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 1 monclient: found mon.noname-a 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93280af980 con 0x7f932813f9a0 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.577+0000 7f932ce77640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ce77640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ce77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9324001690 con 0x7f932813f9a0 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ee7b640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:24:23.562 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ee7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ce77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ce77640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ce77640 10 monclient: dump: 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:23.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.578+0000 7f932ee7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93240018b0 con 0x7f932813f9a0 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ce77640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ce77640 1 -- 172.21.15.138:0/3797073173 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9324001c40 con 0x7f932813f9a0 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ce77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ce77640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ce77640 10 monclient: dump: 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:23.564 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:23.565 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:23.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ee7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.579+0000 7f932ee7b640 4 set_mon_vals no callback set 2024-03-29T11:24:23.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.580+0000 7f932ee7b640 10 monclient: shutdown 2024-03-29T11:24:23.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.580+0000 7f932ee7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.580+0000 7f932ee7b640 1 -- 172.21.15.138:0/3797073173 >> v1:172.21.15.138:6789/0 conn(0x7f932813f9a0 legacy=0x7f932814d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.580+0000 7f932ee7b640 1 -- 172.21.15.138:0/3797073173 shutdown_connections 2024-03-29T11:24:23.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.580+0000 7f932ee7b640 1 -- 172.21.15.138:0/3797073173 >> 172.21.15.138:0/3797073173 conn(0x7f932806d230 msgr2=0x7f9328066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.581+0000 7f932ee7b640 1 -- 172.21.15.138:0/3797073173 shutdown_connections 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.581+0000 7f932ee7b640 1 -- 172.21.15.138:0/3797073173 wait complete. 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.582+0000 7f932ee7b640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.582+0000 7f932ee7b640 10 monclient: monmap: 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.583+0000 7f932ee7b640 1 librados: starting msgr at 2024-03-29T11:24:23.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.583+0000 7f932ee7b640 1 librados: starting objecter 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.583+0000 7f932ee7b640 1 Processor -- start 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.583+0000 7f932ee7b640 1 -- start start 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.583+0000 7f932ee7b640 1 librados: setting wanted keys 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.583+0000 7f932ee7b640 1 librados: calling monclient init 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.583+0000 7f932ee7b640 10 monclient: init 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.584+0000 7f932ee7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.584+0000 7f932ee7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.584+0000 7f932ee7b640 10 monclient(hunting): picked mon.noname-c con 0x7f93280b4200 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.584+0000 7f932ee7b640 10 monclient(hunting): picked mon.noname-b con 0x7f93280b4920 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.584+0000 7f932ee7b640 10 monclient(hunting): picked mon.noname-a con 0x7f932813f9a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.584+0000 7f932e67a640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f932813f9a0 0x7f93281458d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:52058/0 (socket says 172.21.15.138:52058) 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.584+0000 7f932e67a640 1 -- 172.21.15.138:0/559120683 learned_addr learned my addr 172.21.15.138:0/559120683 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.585+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9328141f10 con 0x7f932813f9a0 2024-03-29T11:24:23.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.586+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9328145fe0 con 0x7f93280b4920 2024-03-29T11:24:23.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.586+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9328146170 con 0x7f93280b4200 2024-03-29T11:24:23.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.586+0000 7f932ee7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.586+0000 7f932ee7b640 10 monclient(hunting): authenticate will time out at 1084.895160s 2024-03-29T11:24:23.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 client.?.objecter ms_handle_connect 0x7f932813f9a0 2024-03-29T11:24:23.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 client.?.objecter ms_handle_connect 0x7f93280b4200 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938254 0 0) 0x7f9328146170 con 0x7f93280b4200 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c004eb0 con 0x7f93280b4200 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36699691 0 0) 0x7f9328141f10 con 0x7f932813f9a0 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:24:23.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9328146170 con 0x7f932813f9a0 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 client.?.objecter ms_handle_connect 0x7f93280b4920 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1962525821 0 0) 0x7f9328146170 con 0x7f932813f9a0 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9328141f10 con 0x7f932813f9a0 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9324001bc0 con 0x7f932813f9a0 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.574 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2445091815 0 0) 0x7f930c004eb0 con 0x7f93280b4200 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9328146170 con 0x7f93280b4200 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9320001cb0 con 0x7f93280b4200 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.589+0000 7f931e7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:23.575 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711928907 0 0) 0x7f9328145fe0 con 0x7f93280b4920 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c006c30 con 0x7f93280b4920 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2094780121 0 0) 0x7f9328146170 con 0x7f93280b4200 2024-03-29T11:24:23.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 >> v1:172.21.15.152:6789/0 conn(0x7f93280b4920 legacy=0x7f93281451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 >> v1:172.21.15.138:6789/0 conn(0x7f932813f9a0 legacy=0x7f93281458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 1 monclient: found mon.c 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93280ad520 con 0x7f93280b4200 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f931e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f932ee7b640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f932ee7b640 10 monclient: _renew_subs 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f932ee7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.590+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93280ad770 con 0x7f93280b4200 2024-03-29T11:24:23.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.591+0000 7f932ee7b640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.591+0000 7f932ee7b640 10 monclient: _renew_subs 2024-03-29T11:24:23.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.591+0000 7f932ee7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:23.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.591+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9328141570 con 0x7f93280b4200 2024-03-29T11:24:23.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.591+0000 7f932ee7b640 1 librados: init done 2024-03-29T11:24:23.579 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f932ee7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f932ee7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f932ee7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932804f730 con 0x7f93280b4200 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9320003970 con 0x7f93280b4200 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f931e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9320001cb0 con 0x7f93280b4200 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f931e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f931e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f931e7fc640 10 monclient: dump: 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.580 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.593+0000 7f931ffff640 4 set_mon_vals no callback set 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.594+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f9320019990 con 0x7f93280b4200 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.595+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f9320083840 con 0x7f93280b4200 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.595+0000 7f931e7fc640 10 client.4403.objecter ms_dispatch 0x7f9328000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.595+0000 7f931e7fc640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.595+0000 7f931e7fc640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.595+0000 7f931e7fc640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.596+0000 7f932d678640 10 monclient: get_auth_request con 0x7f930c06e4e0 auth_method 0 2024-03-29T11:24:23.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.596+0000 7f931e7fc640 10 client.4403.objecter ms_handle_connect 0x7f930c06e4e0 2024-03-29T11:24:23.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.603+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f932004f120 con 0x7f93280b4200 2024-03-29T11:24:23.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.603+0000 7f931e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.603+0000 7f931e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.958+0000 7f932ee7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.958+0000 7f932ee7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.958+0000 7f932ee7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:23.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.958+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9328074e10 con 0x7f93280b4200 2024-03-29T11:24:23.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.959+0000 7f931e7fc640 1 -- 172.21.15.138:0/559120683 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f9320051050 con 0x7f93280b4200 2024-03-29T11:24:23.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.959+0000 7f931e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.959+0000 7f931e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.942 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:23.942 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 10 librados: watch_flush enter 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 10 librados: watch_flush exit 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 >> v1:172.21.15.152:6816/3052585572 conn(0x7f930c06e4e0 legacy=0x7f930c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 10 monclient: shutdown 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 >> v1:172.21.15.138:6790/0 conn(0x7f93280b4200 legacy=0x7f93281d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 shutdown_connections 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.964+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 >> 172.21.15.138:0/559120683 conn(0x7f932806d230 msgr2=0x7f93280b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.965+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 shutdown_connections 2024-03-29T11:24:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.965+0000 7f932ee7b640 1 -- 172.21.15.138:0/559120683 wait complete. 2024-03-29T11:24:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:23.965+0000 7f932ee7b640 1 librados: shutdown 2024-03-29T11:24:23.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:30.770 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:30.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.932+0000 7fb3720ba640 10 monclient: get_monmap_and_config 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.932+0000 7fb3720ba640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.932+0000 7fb3720ba640 10 monclient: monmap: 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:30.933735+0000 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:30.933735+0000 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:30.915 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:30.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.933+0000 7fb3720ba640 1 Processor -- start 2024-03-29T11:24:30.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.933+0000 7fb3720ba640 1 -- start start 2024-03-29T11:24:30.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.934+0000 7fb3720ba640 10 monclient: init 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.934+0000 7fb3720ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 10 monclient(hunting): picked mon.noname-c con 0x7fb36c0b0860 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 10 monclient(hunting): picked mon.noname-a con 0x7fb36c072450 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 10 monclient(hunting): picked mon.noname-b con 0x7fb36c13ee40 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c0ad840 con 0x7fb36c072450 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c13c830 con 0x7fb36c0b0860 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c13c9c0 con 0x7fb36c13ee40 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb3720ba640 10 monclient(hunting): authenticate will time out at 1092.244182s 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36b7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fb36c0b0860 0x7fb36c0acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:35540/0 (socket says 172.21.15.138:35540) 2024-03-29T11:24:30.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36b7fe640 1 -- 172.21.15.138:0/1895308823 learned_addr learned my addr 172.21.15.138:0/1895308823 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536478425 0 0) 0x7fb36c13c9c0 con 0x7fb36c13ee40 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb354003520 con 0x7fb36c13ee40 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573927729 0 0) 0x7fb36c0ad840 con 0x7fb36c072450 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.935+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb354003840 con 0x7fb36c072450 2024-03-29T11:24:30.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383710934 0 0) 0x7fb36c13c830 con 0x7fb36c0b0860 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb354003bd0 con 0x7fb36c0b0860 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868820718 0 0) 0x7fb354003520 con 0x7fb36c13ee40 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 >> v1:172.21.15.152:6789/0 conn(0x7fb36c0b0860 legacy=0x7fb36c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 >> v1:172.21.15.138:6789/0 conn(0x7fb36c072450 legacy=0x7fb36c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c13f970 con 0x7fb36c13ee40 2024-03-29T11:24:30.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb364001920 con 0x7fb36c13ee40 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: dump: 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:30.924 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb3720ba640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb3720ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb364001b00 con 0x7fb36c13ee40 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 1 -- 172.21.15.138:0/1895308823 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb364001920 con 0x7fb36c13ee40 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb36a7fc640 10 monclient: dump: 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.925 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb3720ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.936+0000 7fb3720ba640 4 set_mon_vals no callback set 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.937+0000 7fb3720ba640 10 monclient: shutdown 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.937+0000 7fb3720ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.937+0000 7fb3720ba640 1 -- 172.21.15.138:0/1895308823 >> v1:172.21.15.138:6790/0 conn(0x7fb36c13ee40 legacy=0x7fb36c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.937+0000 7fb3720ba640 1 -- 172.21.15.138:0/1895308823 shutdown_connections 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.937+0000 7fb3720ba640 1 -- 172.21.15.138:0/1895308823 >> 172.21.15.138:0/1895308823 conn(0x7fb36c06b660 msgr2=0x7fb36c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.937+0000 7fb3720ba640 1 -- 172.21.15.138:0/1895308823 shutdown_connections 2024-03-29T11:24:30.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.938+0000 7fb3720ba640 1 -- 172.21.15.138:0/1895308823 wait complete. 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.939+0000 7fb3720ba640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.939+0000 7fb3720ba640 10 monclient: monmap: 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:30.927 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.939+0000 7fb3720ba640 1 librados: starting msgr at 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.939+0000 7fb3720ba640 1 librados: starting objecter 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.939+0000 7fb3720ba640 1 Processor -- start 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.940+0000 7fb3720ba640 1 -- start start 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.940+0000 7fb3720ba640 1 librados: setting wanted keys 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.940+0000 7fb3720ba640 1 librados: calling monclient init 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.940+0000 7fb3720ba640 10 monclient: init 2024-03-29T11:24:30.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 10 monclient(hunting): picked mon.noname-c con 0x7fb36c072450 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 10 monclient(hunting): picked mon.noname-a con 0x7fb36c0b0860 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 10 monclient(hunting): picked mon.noname-b con 0x7fb36c13ee40 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c141d60 con 0x7fb36c0b0860 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c0b1270 con 0x7fb36c13ee40 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c1421f0 con 0x7fb36c072450 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb3720ba640 10 monclient(hunting): authenticate will time out at 1092.250382s 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb36b7fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fb36c072450 0x7fb36c140d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:39320/0 (socket says 172.21.15.138:39320) 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb36b7fe640 1 -- 172.21.15.138:0/2756359026 learned_addr learned my addr 172.21.15.138:0/2756359026 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb353fff640 10 client.?.objecter ms_handle_connect 0x7fb36c072450 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.941+0000 7fb353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 client.?.objecter ms_handle_connect 0x7fb36c13ee40 2024-03-29T11:24:30.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555267285 0 0) 0x7fb36c1421f0 con 0x7fb36c072450 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb358004eb0 con 0x7fb36c072450 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 client.?.objecter ms_handle_connect 0x7fb36c0b0860 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.942+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 853487573 0 0) 0x7fb358004eb0 con 0x7fb36c072450 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb36c1421f0 con 0x7fb36c072450 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb3600019c0 con 0x7fb36c072450 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:30.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): dump: 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679679771 0 0) 0x7fb36c141d60 con 0x7fb36c0b0860 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb358004eb0 con 0x7fb36c0b0860 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410982151 0 0) 0x7fb36c0b1270 con 0x7fb36c13ee40 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb358005810 con 0x7fb36c13ee40 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3895504431 0 0) 0x7fb36c1421f0 con 0x7fb36c072450 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 >> v1:172.21.15.152:6789/0 conn(0x7fb36c13ee40 legacy=0x7fb36c1dd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 >> v1:172.21.15.138:6789/0 conn(0x7fb36c0b0860 legacy=0x7fb36c13d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 monclient: found mon.c 2024-03-29T11:24:30.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c143100 con 0x7fb36c072450 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb353fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 10 monclient: _renew_subs 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c13d840 con 0x7fb36c072450 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 10 monclient: _renew_subs 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:30.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c141920 con 0x7fb36c072450 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.943+0000 7fb3720ba640 1 librados: init done 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb360001c40 con 0x7fb36c072450 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb353fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb360004c30 con 0x7fb36c072450 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb353fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb353fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb369ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb353fff640 10 monclient: dump: 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:30.939 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.944+0000 7fb3720ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.945+0000 7fb3720ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.945+0000 7fb3720ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.945+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c005080 con 0x7fb36c072450 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.946+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fb36001ac00 con 0x7fb36c072450 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.946+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fb360084a70 con 0x7fb36c072450 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.946+0000 7fb353fff640 10 client.4409.objecter ms_dispatch 0x7fb36c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.946+0000 7fb353fff640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.946+0000 7fb353fff640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.946+0000 7fb353fff640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:24:30.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.947+0000 7fb36affd640 10 monclient: get_auth_request con 0x7fb35806e170 auth_method 0 2024-03-29T11:24:30.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.947+0000 7fb353fff640 10 client.4409.objecter ms_handle_connect 0x7fb35806e170 2024-03-29T11:24:30.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.954+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3600502c0 con 0x7fb36c072450 2024-03-29T11:24:30.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.954+0000 7fb353fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:30.954+0000 7fb353fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:31.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.325+0000 7fb3720ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:31.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.325+0000 7fb3720ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:31.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.325+0000 7fb3720ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:31.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.325+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb34c005330 con 0x7fb36c072450 2024-03-29T11:24:31.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.328+0000 7fb353fff640 1 -- 172.21.15.138:0/2756359026 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fb3600502c0 con 0x7fb36c072450 2024-03-29T11:24:31.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.328+0000 7fb353fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:31.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.328+0000 7fb353fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:31.311 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:31.311 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 10 librados: watch_flush enter 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 10 librados: watch_flush exit 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 >> v1:172.21.15.152:6816/3052585572 conn(0x7fb35806e170 legacy=0x7fb358070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 10 monclient: shutdown 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 >> v1:172.21.15.138:6790/0 conn(0x7fb36c072450 legacy=0x7fb36c140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 shutdown_connections 2024-03-29T11:24:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 >> 172.21.15.138:0/2756359026 conn(0x7fb36c06b660 msgr2=0x7fb36c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 shutdown_connections 2024-03-29T11:24:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.332+0000 7fb3720ba640 1 -- 172.21.15.138:0/2756359026 wait complete. 2024-03-29T11:24:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:31.333+0000 7fb3720ba640 1 librados: shutdown 2024-03-29T11:24:31.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:38.137 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:38.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.300+0000 7f7e2c045640 10 monclient: get_monmap_and_config 2024-03-29T11:24:38.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.300+0000 7f7e2c045640 10 monclient: build_initial_monmap 2024-03-29T11:24:38.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.300+0000 7f7e2c045640 10 monclient: monmap: 2024-03-29T11:24:38.283 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:38.283 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:38.283 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:38.302160+0000 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:38.302160+0000 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:38.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.301+0000 7f7e2c045640 1 Processor -- start 2024-03-29T11:24:38.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.302+0000 7f7e2c045640 1 -- start start 2024-03-29T11:24:38.287 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.302+0000 7f7e2c045640 10 monclient: init 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.302+0000 7f7e2c045640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.302+0000 7f7e2c045640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.302+0000 7f7e2c045640 10 monclient(hunting): picked mon.noname-a con 0x7f7e240b4330 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.302+0000 7f7e2c045640 10 monclient(hunting): picked mon.noname-b con 0x7f7e241470b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e295e0640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f7e241470b0 0x7f7e240b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:54166/0 (socket says 172.21.15.138:54166) 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e2c045640 10 monclient(hunting): picked mon.noname-c con 0x7f7e241439b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e2c045640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2406f2a0 con 0x7f7e240b4330 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e2c045640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e24146b90 con 0x7f7e241439b0 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e2c045640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240b2ca0 con 0x7f7e241470b0 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e2c045640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e2c045640 10 monclient(hunting): authenticate will time out at 1099.612592s 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e295e0640 1 -- 172.21.15.138:0/2731660251 learned_addr learned my addr 172.21.15.138:0/2731660251 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:38.288 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494137521 0 0) 0x7f7e240b2ca0 con 0x7f7e241470b0 2024-03-29T11:24:38.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e28ddf640 10 monclient(hunting): _negotiate 2024-03-29T11:24:38.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e28ddf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:38.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e28ddf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:38.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e28ddf640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:24:38.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.303+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e10003520 con 0x7f7e241470b0 2024-03-29T11:24:38.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.304+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2235061048 0 0) 0x7f7e10003520 con 0x7f7e241470b0 2024-03-29T11:24:38.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.304+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 >> v1:172.21.15.152:6789/0 conn(0x7f7e241439b0 legacy=0x7f7e24151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:38.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.306+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 >> v1:172.21.15.138:6789/0 conn(0x7f7e240b4330 legacy=0x7f7e240b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:38.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.307+0000 7f7e28ddf640 10 monclient: _finish_hunting 0 2024-03-29T11:24:38.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.307+0000 7f7e28ddf640 1 monclient: found mon.noname-b 2024-03-29T11:24:38.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.307+0000 7f7e28ddf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.307+0000 7f7e28ddf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.307+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2406e0c0 con 0x7f7e241470b0 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e28ddf640 10 monclient: _finish_auth 0 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e28ddf640 10 monclient: _check_auth_tickets 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e28ddf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7e14001690 con 0x7f7e241470b0 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e28ddf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e28ddf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e28ddf640 10 monclient: dump: 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:38.292 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e2c045640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:24:38.293 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.308+0000 7f7e2c045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:38.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e14001920 con 0x7f7e241470b0 2024-03-29T11:24:38.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e28ddf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:38.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e28ddf640 1 -- 172.21.15.138:0/2731660251 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7e14001cb0 con 0x7f7e241470b0 2024-03-29T11:24:38.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e28ddf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e28ddf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e28ddf640 10 monclient: dump: 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:38.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e2c045640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e2c045640 4 set_mon_vals no callback set 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e2c045640 10 monclient: shutdown 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.310+0000 7f7e2c045640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.311+0000 7f7e2c045640 1 -- 172.21.15.138:0/2731660251 >> v1:172.21.15.138:6790/0 conn(0x7f7e241470b0 legacy=0x7f7e240b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.311+0000 7f7e2c045640 1 -- 172.21.15.138:0/2731660251 shutdown_connections 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.311+0000 7f7e2c045640 1 -- 172.21.15.138:0/2731660251 >> 172.21.15.138:0/2731660251 conn(0x7f7e24066a00 msgr2=0x7f7e24066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.311+0000 7f7e2c045640 1 -- 172.21.15.138:0/2731660251 shutdown_connections 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.311+0000 7f7e2c045640 1 -- 172.21.15.138:0/2731660251 wait complete. 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.312+0000 7f7e2c045640 10 monclient: build_initial_monmap 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.312+0000 7f7e2c045640 10 monclient: monmap: 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:38.298 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.312+0000 7f7e2c045640 1 librados: starting msgr at 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.312+0000 7f7e2c045640 1 librados: starting objecter 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.312+0000 7f7e2c045640 1 Processor -- start 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.313+0000 7f7e2c045640 1 -- start start 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.313+0000 7f7e2c045640 1 librados: setting wanted keys 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.313+0000 7f7e2c045640 1 librados: calling monclient init 2024-03-29T11:24:38.299 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.313+0000 7f7e2c045640 10 monclient: init 2024-03-29T11:24:38.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.313+0000 7f7e2c045640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:38.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.313+0000 7f7e2c045640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:38.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.313+0000 7f7e2c045640 10 monclient(hunting): picked mon.noname-c con 0x7f7e240b4330 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.316+0000 7f7e295e0640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f7e241439b0 0x7f7e240a94e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34147/0 (socket says 172.21.15.138:34147) 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.317+0000 7f7e29de1640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f7e240b4330 0x7f7e24141750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:54654/0 (socket says 172.21.15.138:54654) 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.317+0000 7f7e295e0640 1 -- 172.21.15.138:0/3758398114 learned_addr learned my addr 172.21.15.138:0/3758398114 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.317+0000 7f7e2c045640 10 monclient(hunting): picked mon.noname-a con 0x7f7e241439b0 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.317+0000 7f7e1a7fc640 10 client.?.objecter ms_handle_connect 0x7f7e241439b0 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.317+0000 7f7e1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.317+0000 7f7e1a7fc640 10 client.?.objecter ms_handle_connect 0x7f7e240b4330 2024-03-29T11:24:38.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.317+0000 7f7e1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:38.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.318+0000 7f7e2c045640 10 monclient(hunting): picked mon.noname-b con 0x7f7e241470b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:38.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.318+0000 7f7e2c045640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240a9bf0 con 0x7f7e241439b0 2024-03-29T11:24:38.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.318+0000 7f7e2c045640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240aac00 con 0x7f7e241470b0 2024-03-29T11:24:38.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.318+0000 7f7e2c045640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e241458a0 con 0x7f7e240b4330 2024-03-29T11:24:38.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.318+0000 7f7e2c045640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:38.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.318+0000 7f7e2c045640 10 monclient(hunting): authenticate will time out at 1099.627281s 2024-03-29T11:24:38.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.320+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052564806 0 0) 0x7f7e240a9bf0 con 0x7f7e241439b0 2024-03-29T11:24:38.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.320+0000 7f7e1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:38.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.320+0000 7f7e1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:38.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.320+0000 7f7e1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:38.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.320+0000 7f7e1a7fc640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:24:38.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.320+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e08004eb0 con 0x7f7e241439b0 2024-03-29T11:24:38.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 10 client.?.objecter ms_handle_connect 0x7f7e241470b0 2024-03-29T11:24:38.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:38.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389942862 0 0) 0x7f7e08004eb0 con 0x7f7e241439b0 2024-03-29T11:24:38.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e240a9bf0 con 0x7f7e241439b0 2024-03-29T11:24:38.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7e14001610 con 0x7f7e241439b0 2024-03-29T11:24:38.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.306 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1644271329 0 0) 0x7f7e240a9bf0 con 0x7f7e241439b0 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 >> v1:172.21.15.138:6790/0 conn(0x7f7e240b4330 legacy=0x7f7e24141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.307 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.321+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 >> v1:172.21.15.152:6789/0 conn(0x7f7e241470b0 legacy=0x7f7e241ddd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 1 monclient: found mon.a 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e241462f0 con 0x7f7e241439b0 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 10 monclient: _renew_subs 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e24146540 con 0x7f7e241439b0 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 10 monclient: _renew_subs 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:38.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e24144530 con 0x7f7e241439b0 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.322+0000 7f7e2c045640 1 librados: init done 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.323+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e14001770 con 0x7f7e241439b0 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.323+0000 7f7e1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.323+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7e14015e20 con 0x7f7e241439b0 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.323+0000 7f7e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.323+0000 7f7e1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.323+0000 7f7e1a7fc640 10 monclient: dump: 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:38.309 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:38.310 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:38.310 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:38.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.323+0000 7f7e1b7fe640 4 set_mon_vals no callback set 2024-03-29T11:24:38.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.324+0000 7f7e07fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:38.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.326+0000 7f7e07fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:38.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.326+0000 7f7e07fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:38.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.326+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e00005080 con 0x7f7e241439b0 2024-03-29T11:24:38.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.328+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f7e14016040 con 0x7f7e241439b0 2024-03-29T11:24:38.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.329+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f7e14095d60 con 0x7f7e241439b0 2024-03-29T11:24:38.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.329+0000 7f7e1a7fc640 10 client.4398.objecter ms_dispatch 0x7f7e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:38.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.329+0000 7f7e1a7fc640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:38.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.329+0000 7f7e1a7fc640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:38.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.330+0000 7f7e1a7fc640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:24:38.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.331+0000 7f7e29de1640 10 monclient: get_auth_request con 0x7f7e0806dd20 auth_method 0 2024-03-29T11:24:38.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.331+0000 7f7e1a7fc640 10 client.4398.objecter ms_handle_connect 0x7f7e0806dd20 2024-03-29T11:24:38.322 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.339+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e14061640 con 0x7f7e241439b0 2024-03-29T11:24:38.322 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.339+0000 7f7e1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:38.322 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.339+0000 7f7e1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:38.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.663+0000 7f7e07fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.663+0000 7f7e07fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.663+0000 7f7e07fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:24:38.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.663+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e00005330 con 0x7f7e241439b0 2024-03-29T11:24:38.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.664+0000 7f7e1a7fc640 1 -- 172.21.15.138:0/3758398114 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f7e14063570 con 0x7f7e241439b0 2024-03-29T11:24:38.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.664+0000 7f7e1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.664+0000 7f7e1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:38.647 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:38.647 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:38.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.667+0000 7f7e07fff640 10 librados: watch_flush enter 2024-03-29T11:24:38.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.667+0000 7f7e07fff640 10 librados: watch_flush exit 2024-03-29T11:24:38.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.667+0000 7f7e07fff640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:24:38.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.667+0000 7f7e07fff640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:24:38.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.668+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 >> v1:172.21.15.152:6816/3052585572 conn(0x7f7e0806dd20 legacy=0x7f7e080701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.668+0000 7f7e07fff640 10 monclient: shutdown 2024-03-29T11:24:38.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.668+0000 7f7e07fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:38.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.669+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 >> v1:172.21.15.138:6789/0 conn(0x7f7e241439b0 legacy=0x7f7e240a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.669+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 shutdown_connections 2024-03-29T11:24:38.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.669+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 >> 172.21.15.138:0/3758398114 conn(0x7f7e24066a00 msgr2=0x7f7e24070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:38.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.669+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 shutdown_connections 2024-03-29T11:24:38.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.671+0000 7f7e07fff640 1 -- 172.21.15.138:0/3758398114 wait complete. 2024-03-29T11:24:38.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:38.671+0000 7f7e07fff640 1 librados: shutdown 2024-03-29T11:24:38.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:45.478 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.645+0000 7fbc3a9a4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.645+0000 7fbc3a9a4640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.645+0000 7fbc3a9a4640 10 monclient: monmap: 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:45.646320+0000 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:45.646320+0000 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:45.628 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:45.629 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:45.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.646+0000 7fbc3a9a4640 1 Processor -- start 2024-03-29T11:24:45.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.646+0000 7fbc3a9a4640 1 -- start start 2024-03-29T11:24:45.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.646+0000 7fbc3a9a4640 10 monclient: init 2024-03-29T11:24:45.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.648+0000 7fbc3a9a4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.648+0000 7fbc3a9a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:45.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.648+0000 7fbc3a9a4640 10 monclient(hunting): picked mon.noname-b con 0x7fbc2c006890 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:45.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.648+0000 7fbc3a9a4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc2c006c70 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:45.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.648+0000 7fbc3a9a4640 10 monclient(hunting): picked mon.noname-a con 0x7fbc2c02b4c0 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:45.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.650+0000 7fbc3a9a4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c02c830 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.650+0000 7fbc3a9a4640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c02afa0 con 0x7fbc2c006c70 2024-03-29T11:24:45.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.650+0000 7fbc3a9a4640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c02aba0 con 0x7fbc2c006890 2024-03-29T11:24:45.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.650+0000 7fbc3a9a4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.650+0000 7fbc3a9a4640 10 monclient(hunting): authenticate will time out at 1106.959117s 2024-03-29T11:24:45.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.651+0000 7fbc33fff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fbc2c006890 0x7fbc2c0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:45314/0 (socket says 172.21.15.138:45314) 2024-03-29T11:24:45.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.651+0000 7fbc33fff640 1 -- 172.21.15.138:0/32336125 learned_addr learned my addr 172.21.15.138:0/32336125 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:45.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501260971 0 0) 0x7fbc2c02c830 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc24003520 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335250538 0 0) 0x7fbc2c02aba0 con 0x7fbc2c006890 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:24:45.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c02c830 con 0x7fbc2c006890 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432716282 0 0) 0x7fbc2c02c830 con 0x7fbc2c006890 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 >> v1:172.21.15.152:6789/0 conn(0x7fbc2c006c70 legacy=0x7fbc2c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 >> v1:172.21.15.138:6789/0 conn(0x7fbc2c02b4c0 legacy=0x7fbc2c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 monclient: found mon.noname-b 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c02ad30 con 0x7fbc2c006890 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fbc20001610 con 0x7fbc2c006890 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.652+0000 7fbc32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 10 monclient: dump: 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:45.639 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc20001830 con 0x7fbc2c006890 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 1 -- 172.21.15.138:0/32336125 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fbc20001bc0 con 0x7fbc2c006890 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc32ffd640 10 monclient: dump: 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:45.640 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc3a9a4640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc3a9a4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.653+0000 7fbc3a9a4640 4 set_mon_vals no callback set 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.654+0000 7fbc3a9a4640 10 monclient: shutdown 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.654+0000 7fbc3a9a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.654+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/32336125 >> v1:172.21.15.138:6790/0 conn(0x7fbc2c006890 legacy=0x7fbc2c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.654+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/32336125 shutdown_connections 2024-03-29T11:24:45.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.654+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/32336125 >> 172.21.15.138:0/32336125 conn(0x7fbc2c0058f0 msgr2=0x7fbc2c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.656+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/32336125 shutdown_connections 2024-03-29T11:24:45.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.656+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/32336125 wait complete. 2024-03-29T11:24:45.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.657+0000 7fbc3a9a4640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.657+0000 7fbc3a9a4640 10 monclient: monmap: 2024-03-29T11:24:45.643 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.658+0000 7fbc3a9a4640 1 librados: starting msgr at 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.658+0000 7fbc3a9a4640 1 librados: starting objecter 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.658+0000 7fbc3a9a4640 1 Processor -- start 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.659+0000 7fbc3a9a4640 1 -- start start 2024-03-29T11:24:45.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.659+0000 7fbc3a9a4640 1 librados: setting wanted keys 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.659+0000 7fbc3a9a4640 1 librados: calling monclient init 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.659+0000 7fbc3a9a4640 10 monclient: init 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.660+0000 7fbc3a9a4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.660+0000 7fbc3a9a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.660+0000 7fbc3a9a4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc2c006890 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.660+0000 7fbc33fff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fbc2c006890 0x7fbc2c15a220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:45734/0 (socket says 172.21.15.138:45734) 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.660+0000 7fbc33fff640 1 -- 172.21.15.138:0/1321844777 learned_addr learned my addr 172.21.15.138:0/1321844777 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.660+0000 7fbc3a9a4640 10 monclient(hunting): picked mon.noname-b con 0x7fbc2c02b4c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:45.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.660+0000 7fbc3a9a4640 10 monclient(hunting): picked mon.noname-a con 0x7fbc2c02e0a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:45.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.661+0000 7fbc30ff9640 10 client.?.objecter ms_handle_connect 0x7fbc2c006890 2024-03-29T11:24:45.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.663+0000 7fbc30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c1599d0 con 0x7fbc2c02e0a0 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 client.?.objecter ms_handle_connect 0x7fbc2c02e0a0 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 client.?.objecter ms_handle_connect 0x7fbc2c02b4c0 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c157440 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c1575d0 con 0x7fbc2c006890 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc3a9a4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc3a9a4640 10 monclient(hunting): authenticate will time out at 1106.973618s 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242712075 0 0) 0x7fbc2c157440 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:24:45.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc18004eb0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576118581 0 0) 0x7fbc2c1599d0 con 0x7fbc2c02e0a0 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c157440 con 0x7fbc2c02e0a0 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982053216 0 0) 0x7fbc2c1575d0 con 0x7fbc2c006890 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.664+0000 7fbc30ff9640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc18005360 con 0x7fbc2c006890 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2808234180 0 0) 0x7fbc18004eb0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc2c1575d0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fbc28001770 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:45.653 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2943297968 0 0) 0x7fbc2c157440 con 0x7fbc2c02e0a0 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc18004eb0 con 0x7fbc2c02e0a0 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fbc3404fdd0 con 0x7fbc2c02e0a0 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:45.654 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2067278414 0 0) 0x7fbc2c1575d0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 >> v1:172.21.15.138:6790/0 conn(0x7fbc2c006890 legacy=0x7fbc2c15a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 >> v1:172.21.15.138:6789/0 conn(0x7fbc2c02e0a0 legacy=0x7fbc2c156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 monclient: found mon.b 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:45.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c02cdc0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc28001c80 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fbc280037b0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc30ff9640 10 monclient: dump: 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:45.656 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 10 monclient: _renew_subs 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c02cf50 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 10 monclient: _renew_subs 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c02dcb0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.665+0000 7fbc3a9a4640 1 librados: init done 2024-03-29T11:24:45.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.666+0000 7fbc327fc640 4 set_mon_vals no callback set 2024-03-29T11:24:45.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.668+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fbc28019780 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.668+0000 7fbc3a9a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.668+0000 7fbc3a9a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.668+0000 7fbc3a9a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:45.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.668+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc14005080 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.669+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fbc28083770 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.669+0000 7fbc30ff9640 10 client.4363.objecter ms_dispatch 0x7fbc34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.669+0000 7fbc30ff9640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.669+0000 7fbc30ff9640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.669+0000 7fbc30ff9640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.669+0000 7fbc33fff640 10 monclient: get_auth_request con 0x7fbc18076800 auth_method 0 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.670+0000 7fbc30ff9640 10 client.4363.objecter ms_handle_connect 0x7fbc18076800 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.676+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc2804f050 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.676+0000 7fbc30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:45.676+0000 7fbc30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:45.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.004+0000 7fbc3a9a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.004+0000 7fbc3a9a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.004+0000 7fbc3a9a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:24:45.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.004+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc14005330 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.005+0000 7fbc30ff9640 1 -- 172.21.15.138:0/1321844777 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fbc28056ba0 con 0x7fbc2c02b4c0 2024-03-29T11:24:45.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.005+0000 7fbc30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.005+0000 7fbc30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:45.990 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:45.991 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:45.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.010+0000 7fbc3a9a4640 10 librados: watch_flush enter 2024-03-29T11:24:45.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.012+0000 7fbc3a9a4640 10 librados: watch_flush exit 2024-03-29T11:24:45.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.012+0000 7fbc3a9a4640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T11:24:45.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.012+0000 7fbc3a9a4640 10 client.4363.objecter successfully canceled tick 2024-03-29T11:24:45.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.012+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 >> v1:172.21.15.152:6816/3052585572 conn(0x7fbc18076800 legacy=0x7fbc18078cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.012+0000 7fbc3a9a4640 10 monclient: shutdown 2024-03-29T11:24:45.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.012+0000 7fbc3a9a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.012+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 >> v1:172.21.15.152:6789/0 conn(0x7fbc2c02b4c0 legacy=0x7fbc2c156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.013+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 shutdown_connections 2024-03-29T11:24:45.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.013+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 >> 172.21.15.138:0/1321844777 conn(0x7fbc2c0058f0 msgr2=0x7fbc2c0c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.014+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 shutdown_connections 2024-03-29T11:24:45.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.014+0000 7fbc3a9a4640 1 -- 172.21.15.138:0/1321844777 wait complete. 2024-03-29T11:24:45.998 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:46.015+0000 7fbc3a9a4640 1 librados: shutdown 2024-03-29T11:24:46.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:52.823 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:52.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.989+0000 7f6444f32640 10 monclient: get_monmap_and_config 2024-03-29T11:24:52.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.989+0000 7f6444f32640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.989+0000 7f6444f32640 10 monclient: monmap: 2024-03-29T11:24:52.972 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:52.972 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:52.972 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:24:52.990152+0000 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:24:52.990152+0000 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.990+0000 7f6444f32640 1 Processor -- start 2024-03-29T11:24:52.973 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.990+0000 7f6444f32640 1 -- start start 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.990+0000 7f6444f32640 10 monclient: init 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 10 monclient(hunting): picked mon.noname-a con 0x7f64400a9930 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 10 monclient(hunting): picked mon.noname-c con 0x7f6440149dc0 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 10 monclient(hunting): picked mon.noname-b con 0x7f644014ff40 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400aae60 con 0x7f64400a9930 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400b4310 con 0x7f6440149dc0 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400afdd0 con 0x7f644014ff40 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.991+0000 7f6444f32640 10 monclient(hunting): authenticate will time out at 1114.299924s 2024-03-29T11:24:52.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.994+0000 7f643ed9d640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f644014ff40 0x7f6440152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:55822/0 (socket says 172.21.15.138:55822) 2024-03-29T11:24:52.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.994+0000 7f643ed9d640 1 -- 172.21.15.138:0/2364193995 learned_addr learned my addr 172.21.15.138:0/2364193995 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177042260 0 0) 0x7f64400b4310 con 0x7f6440149dc0 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6428003520 con 0x7f6440149dc0 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930642414 0 0) 0x7f64400aae60 con 0x7f64400a9930 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64400b4310 con 0x7f64400a9930 2024-03-29T11:24:52.981 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707201350 0 0) 0x7f64400b4310 con 0x7f64400a9930 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 >> v1:172.21.15.138:6790/0 conn(0x7f644014ff40 legacy=0x7f6440152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 >> v1:172.21.15.152:6789/0 conn(0x7f6440149dc0 legacy=0x7f644014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 monclient: found mon.noname-a 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644014a950 con 0x7f64400a9930 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6434001670 con 0x7f64400a9930 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:52.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: dump: 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f6444f32640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f6444f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 1 -- 172.21.15.138:0/2364193995 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64340017d0 con 0x7f64400a9930 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f643d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f6444f32640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.996+0000 7f6444f32640 4 set_mon_vals no callback set 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.997+0000 7f6444f32640 10 monclient: shutdown 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.997+0000 7f6444f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.997+0000 7f6444f32640 1 -- 172.21.15.138:0/2364193995 >> v1:172.21.15.138:6789/0 conn(0x7f64400a9930 legacy=0x7f6440143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.998+0000 7f6444f32640 1 -- 172.21.15.138:0/2364193995 shutdown_connections 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.998+0000 7f6444f32640 1 -- 172.21.15.138:0/2364193995 >> 172.21.15.138:0/2364193995 conn(0x7f64400b35e0 msgr2=0x7f64400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.998+0000 7f6444f32640 1 -- 172.21.15.138:0/2364193995 shutdown_connections 2024-03-29T11:24:52.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:52.998+0000 7f6444f32640 1 -- 172.21.15.138:0/2364193995 wait complete. 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.000+0000 7f6444f32640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.000+0000 7f6444f32640 10 monclient: monmap: 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:24:52.985 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.000+0000 7f6444f32640 1 librados: starting msgr at 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.000+0000 7f6444f32640 1 librados: starting objecter 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.000+0000 7f6444f32640 1 Processor -- start 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.001+0000 7f6444f32640 1 -- start start 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.001+0000 7f6444f32640 1 librados: setting wanted keys 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.001+0000 7f6444f32640 1 librados: calling monclient init 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.001+0000 7f6444f32640 10 monclient: init 2024-03-29T11:24:52.986 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 10 monclient(hunting): picked mon.noname-a con 0x7f64400a9930 addr v1:172.21.15.138:6789/0 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 10 monclient(hunting): picked mon.noname-b con 0x7f6440149dc0 addr v1:172.21.15.152:6789/0 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 10 monclient(hunting): picked mon.noname-c con 0x7f644014ff40 addr v1:172.21.15.138:6790/0 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400acf30 con 0x7f64400a9930 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400acae0 con 0x7f6440149dc0 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f643ed9d640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f644014ff40 0x7f64401dbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:56050/0 (socket says 172.21.15.138:56050) 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f643ed9d640 1 -- 172.21.15.138:0/2221143453 learned_addr learned my addr 172.21.15.138:0/2221143453 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400acc70 con 0x7f644014ff40 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f6444f32640 10 monclient(hunting): authenticate will time out at 1114.310597s 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f642effd640 10 client.?.objecter ms_handle_connect 0x7f644014ff40 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.002+0000 7f642effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.987 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153292976 0 0) 0x7f64400acc70 con 0x7f644014ff40 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6420004eb0 con 0x7f644014ff40 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1045012286 0 0) 0x7f6420004eb0 con 0x7f644014ff40 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64400acc70 con 0x7f644014ff40 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f64300017d0 con 0x7f644014ff40 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): dump: 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:52.988 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.003+0000 7f642effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.006+0000 7f642effd640 10 client.?.objecter ms_handle_connect 0x7f6440149dc0 2024-03-29T11:24:52.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.006+0000 7f642effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.006+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2288550151 0 0) 0x7f64400acc70 con 0x7f644014ff40 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.006+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 >> v1:172.21.15.152:6789/0 conn(0x7f6440149dc0 legacy=0x7f64401db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.006+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 >> v1:172.21.15.138:6789/0 conn(0x7f64400a9930 legacy=0x7f6440144510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 1 monclient: found mon.c 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64401488c0 con 0x7f644014ff40 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f642effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 10 monclient: _renew_subs 2024-03-29T11:24:52.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:52.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6440148b70 con 0x7f644014ff40 2024-03-29T11:24:52.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:52.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 10 monclient: _renew_subs 2024-03-29T11:24:52.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:52.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64401e7190 con 0x7f644014ff40 2024-03-29T11:24:52.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.007+0000 7f6444f32640 1 librados: init done 2024-03-29T11:24:52.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.010+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6430001930 con 0x7f644014ff40 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.010+0000 7f642effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.010+0000 7f642ffff640 4 set_mon_vals no callback set 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f64300049c0 con 0x7f644014ff40 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 10 monclient: dump: 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:24:52.993 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:24:52.994 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f6430004be0 con 0x7f644014ff40 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f6444f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f6444f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f6444f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6424005080 con 0x7f644014ff40 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f6430084a50 con 0x7f644014ff40 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 10 client.4424.objecter ms_dispatch 0x7f6440000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:52.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.011+0000 7f642effd640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:52.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.012+0000 7f642effd640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:24:52.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.012+0000 7f643e59c640 10 monclient: get_auth_request con 0x7f642006de10 auth_method 0 2024-03-29T11:24:52.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.013+0000 7f642effd640 10 client.4424.objecter ms_handle_connect 0x7f642006de10 2024-03-29T11:24:53.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.021+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64300503b0 con 0x7f644014ff40 2024-03-29T11:24:53.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.021+0000 7f642effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:53.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.021+0000 7f642effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:53.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.357+0000 7f6444f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:53.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.357+0000 7f6444f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:53.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.357+0000 7f6444f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:24:53.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.357+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6424005330 con 0x7f644014ff40 2024-03-29T11:24:53.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.357+0000 7f642effd640 1 -- 172.21.15.138:0/2221143453 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f64300522e0 con 0x7f644014ff40 2024-03-29T11:24:53.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.358+0000 7f642effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:53.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.358+0000 7f642effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:53.340 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:24:53.340 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:53.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 10 librados: watch_flush enter 2024-03-29T11:24:53.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 10 librados: watch_flush exit 2024-03-29T11:24:53.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:24:53.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:24:53.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 >> v1:172.21.15.152:6816/3052585572 conn(0x7f642006de10 legacy=0x7f64200702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 10 monclient: shutdown 2024-03-29T11:24:53.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:53.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.361+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 >> v1:172.21.15.138:6790/0 conn(0x7f644014ff40 legacy=0x7f64401dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.362+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 shutdown_connections 2024-03-29T11:24:53.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.362+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 >> 172.21.15.138:0/2221143453 conn(0x7f64400b35e0 msgr2=0x7f6440143800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:53.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.362+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 shutdown_connections 2024-03-29T11:24:53.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.362+0000 7f6444f32640 1 -- 172.21.15.138:0/2221143453 wait complete. 2024-03-29T11:24:53.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:24:53.362+0000 7f6444f32640 1 librados: shutdown 2024-03-29T11:24:53.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:00.169 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.337+0000 7f8398155640 10 monclient: get_monmap_and_config 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.337+0000 7f8398155640 10 monclient: build_initial_monmap 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.337+0000 7f8398155640 10 monclient: monmap: 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:00.338940+0000 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:00.338940+0000 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:00.321 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:00.322 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:00.322 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:00.322 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:00.322 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:00.322 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.338+0000 7f8398155640 1 Processor -- start 2024-03-29T11:25:00.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 1 -- start start 2024-03-29T11:25:00.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient: init 2024-03-29T11:25:00.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient(hunting): picked mon.noname-c con 0x7f83900b46b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient(hunting): picked mon.noname-b con 0x7f83900b3da0 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient(hunting): picked mon.noname-a con 0x7f8390142a40 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83900b3530 con 0x7f8390142a40 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83900b4280 con 0x7f83900b46b0 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83900aa9f0 con 0x7f83900b3da0 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:00.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.339+0000 7f8398155640 10 monclient(hunting): authenticate will time out at 1121.649036s 2024-03-29T11:25:00.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.341+0000 7f83956f0640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f83900b3da0 0x7f839013b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:46036/0 (socket says 172.21.15.138:46036) 2024-03-29T11:25:00.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.341+0000 7f83956f0640 1 -- 172.21.15.138:0/1667071477 learned_addr learned my addr 172.21.15.138:0/1667071477 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.343+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661154938 0 0) 0x7f83900aa9f0 con 0x7f83900b3da0 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.343+0000 7f8394eef640 10 monclient(hunting): _negotiate 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.343+0000 7f8394eef640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.343+0000 7f8394eef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.343+0000 7f8394eef640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.343+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f837c003520 con 0x7f83900b3da0 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 534702638 0 0) 0x7f837c003520 con 0x7f83900b3da0 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 >> v1:172.21.15.152:6789/0 conn(0x7f83900b46b0 legacy=0x7f8390142330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 >> v1:172.21.15.138:6789/0 conn(0x7f8390142a40 legacy=0x7f839014d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 10 monclient: _finish_hunting 0 2024-03-29T11:25:00.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 1 monclient: found mon.noname-b 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83900aacd0 con 0x7f83900b3da0 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 10 monclient: _finish_auth 0 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 10 monclient: _check_auth_tickets 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8398155640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8398155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f838c001690 con 0x7f83900b3da0 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8394eef640 10 monclient: dump: 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:00.330 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.344+0000 7f8398155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f838c001920 con 0x7f83900b3da0 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8394eef640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8398155640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8398155640 4 set_mon_vals no callback set 2024-03-29T11:25:00.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8394eef640 1 -- 172.21.15.138:0/1667071477 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f838c001cb0 con 0x7f83900b3da0 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8394eef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8394eef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8394eef640 10 monclient: dump: 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:00.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8398155640 10 monclient: shutdown 2024-03-29T11:25:00.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8398155640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:00.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.345+0000 7f8398155640 1 -- 172.21.15.138:0/1667071477 >> v1:172.21.15.138:6790/0 conn(0x7f83900b3da0 legacy=0x7f839013b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.346+0000 7f8398155640 1 -- 172.21.15.138:0/1667071477 shutdown_connections 2024-03-29T11:25:00.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.346+0000 7f8398155640 1 -- 172.21.15.138:0/1667071477 >> 172.21.15.138:0/1667071477 conn(0x7f839006d0f0 msgr2=0x7f8390066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:00.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.348+0000 7f8398155640 1 -- 172.21.15.138:0/1667071477 shutdown_connections 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.349+0000 7f8398155640 1 -- 172.21.15.138:0/1667071477 wait complete. 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 10 monclient: build_initial_monmap 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 10 monclient: monmap: 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:00.334 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:00.335 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 1 librados: starting msgr at 2024-03-29T11:25:00.335 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 1 librados: starting objecter 2024-03-29T11:25:00.335 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 1 Processor -- start 2024-03-29T11:25:00.336 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 1 -- start start 2024-03-29T11:25:00.336 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 1 librados: setting wanted keys 2024-03-29T11:25:00.337 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 1 librados: calling monclient init 2024-03-29T11:25:00.337 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.350+0000 7f8398155640 10 monclient: init 2024-03-29T11:25:00.337 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.351+0000 7f8398155640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:00.337 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.351+0000 7f8398155640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:00.337 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.351+0000 7f8398155640 10 monclient(hunting): picked mon.noname-a con 0x7f83900b3da0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:00.337 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.351+0000 7f8398155640 10 monclient(hunting): picked mon.noname-c con 0x7f83900b46b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:00.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f8398155640 10 monclient(hunting): picked mon.noname-b con 0x7f8390142a40 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:00.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f8398155640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839006f5d0 con 0x7f83900b3da0 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f8398155640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839006f2f0 con 0x7f8390142a40 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f8398155640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839006e810 con 0x7f83900b46b0 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f8398155640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f8398155640 10 monclient(hunting): authenticate will time out at 1121.663125s 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83966f2640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f8390142a40 0x7f83901eeb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:36795/0 (socket says 172.21.15.138:36795) 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83966f2640 1 -- 172.21.15.138:0/4128022815 learned_addr learned my addr 172.21.15.138:0/4128022815 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 10 client.?.objecter ms_handle_connect 0x7f8390142a40 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248242995 0 0) 0x7f839006f2f0 con 0x7f8390142a40 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:00.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.354+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374004eb0 con 0x7f8390142a40 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304930994 0 0) 0x7f8374004eb0 con 0x7f8390142a40 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f839006f2f0 con 0x7f8390142a40 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8388001b00 con 0x7f8390142a40 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 10 monclient(hunting): dump: 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:00.341 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:00.342 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:00.342 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:00.342 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:00.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:00.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3177065645 0 0) 0x7f839006f2f0 con 0x7f8390142a40 2024-03-29T11:25:00.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 >> v1:172.21.15.138:6790/0 conn(0x7f83900b46b0 legacy=0x7f83901e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.355+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 >> v1:172.21.15.138:6789/0 conn(0x7f83900b3da0 legacy=0x7f839006e100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:00.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:00.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 1 monclient: found mon.b 2024-03-29T11:25:00.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:00.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:00.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839013c490 con 0x7f8390142a40 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f83867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f8398155640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f8398155640 10 monclient: _renew_subs 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f8398155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.360+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839013e9c0 con 0x7f8390142a40 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83880020d0 con 0x7f8390142a40 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f83867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8388004db0 con 0x7f8390142a40 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f83867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f83867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f83867fc640 10 monclient: dump: 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:00.345 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f8398155640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f8398155640 10 monclient: _renew_subs 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f8398155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83901e4fa0 con 0x7f8390142a40 2024-03-29T11:25:00.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f8398155640 1 librados: init done 2024-03-29T11:25:00.348 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.361+0000 7f83877fe640 4 set_mon_vals no callback set 2024-03-29T11:25:00.348 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.362+0000 7f8398155640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.363+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f838801ad80 con 0x7f8390142a40 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.363+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f8388084da0 con 0x7f8390142a40 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.363+0000 7f83867fc640 10 client.4372.objecter ms_dispatch 0x7f8390000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.363+0000 7f83867fc640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.363+0000 7f83867fc640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.363+0000 7f83867fc640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.364+0000 7f8398155640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.364+0000 7f8398155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.364+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8378005080 con 0x7f8390142a40 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.364+0000 7f8395ef1640 10 monclient: get_auth_request con 0x7f837406dc40 auth_method 0 2024-03-29T11:25:00.349 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.364+0000 7f83867fc640 10 client.4372.objecter ms_handle_connect 0x7f837406dc40 2024-03-29T11:25:00.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.371+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83880504b0 con 0x7f8390142a40 2024-03-29T11:25:00.355 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.371+0000 7f83867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:00.355 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.371+0000 7f83867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:00.693 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.710+0000 7f8398155640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.693 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.710+0000 7f8398155640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.693 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.710+0000 7f8398155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:00.693 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.710+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8378005330 con 0x7f8390142a40 2024-03-29T11:25:00.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.711+0000 7f83867fc640 1 -- 172.21.15.138:0/4128022815 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f83880525b0 con 0x7f8390142a40 2024-03-29T11:25:00.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.711+0000 7f83867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.711+0000 7f83867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:00.696 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:00.696 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:00.700 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.716+0000 7f8398155640 10 librados: watch_flush enter 2024-03-29T11:25:00.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.718+0000 7f8398155640 10 librados: watch_flush exit 2024-03-29T11:25:00.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.718+0000 7f8398155640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-29T11:25:00.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.718+0000 7f8398155640 10 client.4372.objecter successfully canceled tick 2024-03-29T11:25:00.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.719+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 >> v1:172.21.15.152:6816/3052585572 conn(0x7f837406dc40 legacy=0x7f8374070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.719+0000 7f8398155640 10 monclient: shutdown 2024-03-29T11:25:00.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.719+0000 7f8398155640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:00.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.719+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 >> v1:172.21.15.152:6789/0 conn(0x7f8390142a40 legacy=0x7f83901eeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.719+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 shutdown_connections 2024-03-29T11:25:00.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.719+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 >> 172.21.15.138:0/4128022815 conn(0x7f839006d0f0 msgr2=0x7f83900b8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:00.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.719+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 shutdown_connections 2024-03-29T11:25:00.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.720+0000 7f8398155640 1 -- 172.21.15.138:0/4128022815 wait complete. 2024-03-29T11:25:00.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:00.720+0000 7f8398155640 1 librados: shutdown 2024-03-29T11:25:00.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:07.528 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:07.679 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.695+0000 7f0197fff640 10 monclient: get_monmap_and_config 2024-03-29T11:25:07.679 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.695+0000 7f0197fff640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.695+0000 7f0197fff640 10 monclient: monmap: 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:07.697178+0000 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:07.697178+0000 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:07.680 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:07.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.696+0000 7f0197fff640 1 Processor -- start 2024-03-29T11:25:07.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.696+0000 7f0197fff640 1 -- start start 2024-03-29T11:25:07.681 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.696+0000 7f0197fff640 10 monclient: init 2024-03-29T11:25:07.683 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 10 monclient(hunting): picked mon.noname-c con 0x7f01900389e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0190038dc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0190030660 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01900c54c0 con 0x7f0190038dc0 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0190036430 con 0x7f01900389e0 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0190039840 con 0x7f0190030660 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f0197fff640 10 monclient(hunting): authenticate will time out at 1129.006775s 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f01977fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f0190030660 0x7f01900d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:55206/0 (socket says 172.21.15.138:55206) 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.697+0000 7f01977fe640 1 -- 172.21.15.138:0/4208472430 learned_addr learned my addr 172.21.15.138:0/4208472430 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.700+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088177189 0 0) 0x7f0190039840 con 0x7f0190030660 2024-03-29T11:25:07.684 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.700+0000 7f0195ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.685 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.700+0000 7f0195ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.685 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.700+0000 7f0195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.685 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.700+0000 7f0195ffb640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:25:07.685 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.700+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180003520 con 0x7f0190030660 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.701+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1315895869 0 0) 0x7f0180003520 con 0x7f0190030660 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.701+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 >> v1:172.21.15.152:6789/0 conn(0x7f01900389e0 legacy=0x7f0190033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 >> v1:172.21.15.138:6789/0 conn(0x7f0190038dc0 legacy=0x7f01900cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 1 monclient: found mon.noname-b 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190035690 con 0x7f0190030660 2024-03-29T11:25:07.687 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0197fff640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0197fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0188001690 con 0x7f0190030660 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0195ffb640 10 monclient: dump: 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:07.688 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:07.689 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:07.689 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:07.689 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:07.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.702+0000 7f0197fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.704+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0188001920 con 0x7f0190030660 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.704+0000 7f0195ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.704+0000 7f0197fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.704+0000 7f0197fff640 4 set_mon_vals no callback set 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.705+0000 7f0195ffb640 1 -- 172.21.15.138:0/4208472430 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0188001cb0 con 0x7f0190030660 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.705+0000 7f0195ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.705+0000 7f0195ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.705+0000 7f0195ffb640 10 monclient: dump: 2024-03-29T11:25:07.690 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.705+0000 7f0197fff640 10 monclient: shutdown 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.705+0000 7f0197fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.705+0000 7f0197fff640 1 -- 172.21.15.138:0/4208472430 >> v1:172.21.15.138:6790/0 conn(0x7f0190030660 legacy=0x7f01900d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.706+0000 7f0197fff640 1 -- 172.21.15.138:0/4208472430 shutdown_connections 2024-03-29T11:25:07.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.706+0000 7f0197fff640 1 -- 172.21.15.138:0/4208472430 >> 172.21.15.138:0/4208472430 conn(0x7f0190005c00 msgr2=0x7f0190006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:07.692 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.709+0000 7f0197fff640 1 -- 172.21.15.138:0/4208472430 shutdown_connections 2024-03-29T11:25:07.693 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.709+0000 7f0197fff640 1 -- 172.21.15.138:0/4208472430 wait complete. 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.710+0000 7f0197fff640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.710+0000 7f0197fff640 10 monclient: monmap: 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:07.697 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.711+0000 7f0197fff640 1 librados: starting msgr at 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.711+0000 7f0197fff640 1 librados: starting objecter 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.711+0000 7f0197fff640 1 Processor -- start 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.711+0000 7f0197fff640 1 -- start start 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.711+0000 7f0197fff640 1 librados: setting wanted keys 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.711+0000 7f0197fff640 1 librados: calling monclient init 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.711+0000 7f0197fff640 10 monclient: init 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0190030660 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 10 monclient(hunting): picked mon.noname-c con 0x7f01900389e0 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 10 monclient(hunting): picked mon.noname-a con 0x7f019002ff90 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:07.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019002e9e0 con 0x7f019002ff90 2024-03-29T11:25:07.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019002f380 con 0x7f0190030660 2024-03-29T11:25:07.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019002dad0 con 0x7f01900389e0 2024-03-29T11:25:07.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.713+0000 7f0197fff640 10 monclient(hunting): authenticate will time out at 1129.022636s 2024-03-29T11:25:07.700 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.717+0000 7f01967fc640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f01900389e0 0x7f019002f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:36404/0 (socket says 172.21.15.138:36404) 2024-03-29T11:25:07.700 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.717+0000 7f01967fc640 1 -- 172.21.15.138:0/278801713 learned_addr learned my addr 172.21.15.138:0/278801713 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:07.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.718+0000 7f017f7fe640 10 client.?.objecter ms_handle_connect 0x7f01900389e0 2024-03-29T11:25:07.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.718+0000 7f017f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.718+0000 7f017f7fe640 10 client.?.objecter ms_handle_connect 0x7f019002ff90 2024-03-29T11:25:07.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.718+0000 7f017f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250573062 0 0) 0x7f019002e9e0 con 0x7f019002ff90 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0184004eb0 con 0x7f019002ff90 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 client.?.objecter ms_handle_connect 0x7f0190030660 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963542647 0 0) 0x7f019002f380 con 0x7f0190030660 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:25:07.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.719+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019002e9e0 con 0x7f0190030660 2024-03-29T11:25:07.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328290279 0 0) 0x7f019002e9e0 con 0x7f0190030660 2024-03-29T11:25:07.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f019002f380 con 0x7f0190030660 2024-03-29T11:25:07.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 245854605 0 0) 0x7f0184004eb0 con 0x7f019002ff90 2024-03-29T11:25:07.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f019002e9e0 con 0x7f019002ff90 2024-03-29T11:25:07.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f019804fdd0 con 0x7f0190030660 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 10 monclient(hunting): dump: 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:07.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f01880013b0 con 0x7f019002ff90 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 10 monclient(hunting): dump: 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:07.706 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:07.707 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.720+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968310857 0 0) 0x7f019002dad0 con 0x7f01900389e0 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0184004eb0 con 0x7f01900389e0 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3359922627 0 0) 0x7f019002f380 con 0x7f0190030660 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 >> v1:172.21.15.138:6790/0 conn(0x7f01900389e0 legacy=0x7f019002f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 >> v1:172.21.15.138:6789/0 conn(0x7f019002ff90 legacy=0x7f01901673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 1 monclient: found mon.b 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019002dc60 con 0x7f0190030660 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f017f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 10 monclient: _renew_subs 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01900c5710 con 0x7f0190030660 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:07.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 10 monclient: _renew_subs 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01900c7df0 con 0x7f0190030660 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.722+0000 7f0197fff640 1 librados: init done 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f019804f400 con 0x7f0190030660 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f017f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f0198075f10 con 0x7f0190030660 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f017f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f017f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f017f7fe640 10 monclient: dump: 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.711 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f01957fa640 4 set_mon_vals no callback set 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.723+0000 7f0197fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.724+0000 7f0197fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.724+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f0198067bf0 con 0x7f0190030660 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.724+0000 7f0197fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.724+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0178005080 con 0x7f0190030660 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.725+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f01980ed140 con 0x7f0190030660 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.725+0000 7f017f7fe640 10 client.4375.objecter ms_dispatch 0x7f0198000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.725+0000 7f017f7fe640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.725+0000 7f017f7fe640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:07.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.726+0000 7f017f7fe640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T11:25:07.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.726+0000 7f01967fc640 10 monclient: get_auth_request con 0x7f018406ddf0 auth_method 0 2024-03-29T11:25:07.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.727+0000 7f017f7fe640 10 client.4375.objecter ms_handle_connect 0x7f018406ddf0 2024-03-29T11:25:07.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.733+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01980b8a80 con 0x7f0190030660 2024-03-29T11:25:07.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.733+0000 7f017f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:07.733+0000 7f017f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:08.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.056+0000 7f0197fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.056+0000 7f0197fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.056+0000 7f0197fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:08.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.056+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0178005330 con 0x7f0190030660 2024-03-29T11:25:08.044 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:08.044 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:08.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.058+0000 7f017f7fe640 1 -- 172.21.15.138:0/278801713 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f01980ec7f0 con 0x7f0190030660 2024-03-29T11:25:08.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.058+0000 7f017f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.058+0000 7f017f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:08.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 10 librados: watch_flush enter 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 10 librados: watch_flush exit 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 10 client.4375.objecter successfully canceled tick 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 >> v1:172.21.15.152:6816/3052585572 conn(0x7f018406ddf0 legacy=0x7f01840702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 10 monclient: shutdown 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.061+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 >> v1:172.21.15.152:6789/0 conn(0x7f0190030660 legacy=0x7f019002d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.062+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 shutdown_connections 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.062+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 >> 172.21.15.138:0/278801713 conn(0x7f0190005c00 msgr2=0x7f0190034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.062+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 shutdown_connections 2024-03-29T11:25:08.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.062+0000 7f0197fff640 1 -- 172.21.15.138:0/278801713 wait complete. 2024-03-29T11:25:08.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:08.062+0000 7f0197fff640 1 librados: shutdown 2024-03-29T11:25:08.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:14.871 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:15.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.036+0000 7fdb47923640 10 monclient: get_monmap_and_config 2024-03-29T11:25:15.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.036+0000 7fdb47923640 10 monclient: build_initial_monmap 2024-03-29T11:25:15.022 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.036+0000 7fdb47923640 10 monclient: monmap: 2024-03-29T11:25:15.022 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:15.022 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:15.023 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:15.038041+0000 2024-03-29T11:25:15.023 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:15.038041+0000 2024-03-29T11:25:15.023 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:15.023 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:15.023 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:15.023 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:15.024 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:15.024 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:15.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.038+0000 7fdb47923640 1 Processor -- start 2024-03-29T11:25:15.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.038+0000 7fdb47923640 1 -- start start 2024-03-29T11:25:15.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.038+0000 7fdb47923640 10 monclient: init 2024-03-29T11:25:15.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:15.027 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:15.028 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 10 monclient(hunting): picked mon.noname-a con 0x7fdb400a9930 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:15.028 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 10 monclient(hunting): picked mon.noname-c con 0x7fdb40149dc0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:15.028 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4014ff40 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:15.028 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400aae60 con 0x7fdb400a9930 2024-03-29T11:25:15.028 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400b4310 con 0x7fdb40149dc0 2024-03-29T11:25:15.028 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400afdd0 con 0x7fdb4014ff40 2024-03-29T11:25:15.029 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:15.029 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.039+0000 7fdb47923640 10 monclient(hunting): authenticate will time out at 1136.348973s 2024-03-29T11:25:15.030 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.043+0000 7fdb45ec0640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fdb4014ff40 0x7fdb40152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:59389/0 (socket says 172.21.15.138:59389) 2024-03-29T11:25:15.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.043+0000 7fdb45ec0640 1 -- 172.21.15.138:0/1059071910 learned_addr learned my addr 172.21.15.138:0/1059071910 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:15.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.048+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487574981 0 0) 0x7fdb400afdd0 con 0x7fdb4014ff40 2024-03-29T11:25:15.031 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.048+0000 7fdb37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.048+0000 7fdb37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.048+0000 7fdb37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.048+0000 7fdb37fff640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:25:15.032 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.048+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2c003520 con 0x7fdb4014ff40 2024-03-29T11:25:15.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.050+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737510198 0 0) 0x7fdb400b4310 con 0x7fdb40149dc0 2024-03-29T11:25:15.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.050+0000 7fdb37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.050+0000 7fdb37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.050+0000 7fdb37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.050+0000 7fdb37fff640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:25:15.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb400afdd0 con 0x7fdb40149dc0 2024-03-29T11:25:15.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3120597428 0 0) 0x7fdb400afdd0 con 0x7fdb40149dc0 2024-03-29T11:25:15.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 >> v1:172.21.15.138:6790/0 conn(0x7fdb4014ff40 legacy=0x7fdb40152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 >> v1:172.21.15.138:6789/0 conn(0x7fdb400a9930 legacy=0x7fdb40143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 monclient: found mon.noname-c 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4014a950 con 0x7fdb40149dc0 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:15.038 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb47923640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb47923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb380019f0 con 0x7fdb40149dc0 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: dump: 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:15.039 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb47923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 1 -- 172.21.15.138:0/1059071910 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb38001c40 con 0x7fdb40149dc0 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb47923640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.051+0000 7fdb47923640 4 set_mon_vals no callback set 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.052+0000 7fdb47923640 10 monclient: shutdown 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.052+0000 7fdb47923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:15.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.052+0000 7fdb47923640 1 -- 172.21.15.138:0/1059071910 >> v1:172.21.15.152:6789/0 conn(0x7fdb40149dc0 legacy=0x7fdb4014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.041 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.055+0000 7fdb47923640 1 -- 172.21.15.138:0/1059071910 shutdown_connections 2024-03-29T11:25:15.041 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.055+0000 7fdb47923640 1 -- 172.21.15.138:0/1059071910 >> 172.21.15.138:0/1059071910 conn(0x7fdb400b35e0 msgr2=0x7fdb400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:15.041 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.057+0000 7fdb47923640 1 -- 172.21.15.138:0/1059071910 shutdown_connections 2024-03-29T11:25:15.041 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.057+0000 7fdb47923640 1 -- 172.21.15.138:0/1059071910 wait complete. 2024-03-29T11:25:15.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 10 monclient: build_initial_monmap 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 10 monclient: monmap: 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 1 librados: starting msgr at 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 1 librados: starting objecter 2024-03-29T11:25:15.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 1 Processor -- start 2024-03-29T11:25:15.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 1 -- start start 2024-03-29T11:25:15.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 1 librados: setting wanted keys 2024-03-29T11:25:15.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.058+0000 7fdb47923640 1 librados: calling monclient init 2024-03-29T11:25:15.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.059+0000 7fdb47923640 10 monclient: init 2024-03-29T11:25:15.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.059+0000 7fdb47923640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:15.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.059+0000 7fdb47923640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:15.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.060+0000 7fdb47923640 10 monclient(hunting): picked mon.noname-c con 0x7fdb400a9930 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:15.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.060+0000 7fdb47923640 10 monclient(hunting): picked mon.noname-b con 0x7fdb40149dc0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:15.050 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.063+0000 7fdb47923640 10 monclient(hunting): picked mon.noname-a con 0x7fdb4014ff40 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:15.050 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.063+0000 7fdb44ebe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fdb40149dc0 0x7fdb401db9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:33903/0 (socket says 172.21.15.138:33903) 2024-03-29T11:25:15.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.063+0000 7fdb44ebe640 1 -- 172.21.15.138:0/1109053478 learned_addr learned my addr 172.21.15.138:0/1109053478 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:15.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.063+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb401db540 con 0x7fdb4014ff40 2024-03-29T11:25:15.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.063+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb40149ad0 con 0x7fdb40149dc0 2024-03-29T11:25:15.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.063+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb401dc0e0 con 0x7fdb400a9930 2024-03-29T11:25:15.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb47923640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:15.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb47923640 10 monclient(hunting): authenticate will time out at 1136.373136s 2024-03-29T11:25:15.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 client.?.objecter ms_handle_connect 0x7fdb40149dc0 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913648959 0 0) 0x7fdb40149ad0 con 0x7fdb40149dc0 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb24004eb0 con 0x7fdb40149dc0 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 client.?.objecter ms_handle_connect 0x7fdb400a9930 2024-03-29T11:25:15.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.064+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 458168393 0 0) 0x7fdb24004eb0 con 0x7fdb40149dc0 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb40149ad0 con 0x7fdb40149dc0 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb38003bd0 con 0x7fdb40149dc0 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:15.053 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:15.054 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:15.054 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.054 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:15.054 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:15.054 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:15.054 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:15.054 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:15.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:15.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103275877 0 0) 0x7fdb40149ad0 con 0x7fdb40149dc0 2024-03-29T11:25:15.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 >> v1:172.21.15.138:6790/0 conn(0x7fdb400a9930 legacy=0x7fdb40147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.065+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 >> v1:172.21.15.138:6789/0 conn(0x7fdb4014ff40 legacy=0x7fdb401e5ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 1 monclient: found mon.b 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb401dc270 con 0x7fdb40149dc0 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.069+0000 7fdb35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 10 monclient: _renew_subs 2024-03-29T11:25:15.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4013e040 con 0x7fdb40149dc0 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 10 monclient: _renew_subs 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb38003df0 con 0x7fdb40149dc0 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb40148ec0 con 0x7fdb40149dc0 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb47923640 1 librados: init done 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.070+0000 7fdb35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.071+0000 7fdb377fe640 4 set_mon_vals no callback set 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.071+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb38004300 con 0x7fdb40149dc0 2024-03-29T11:25:15.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.072+0000 7fdb35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.072+0000 7fdb35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.072+0000 7fdb35ffb640 10 monclient: dump: 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:15.062 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.073+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fdb38001e30 con 0x7fdb40149dc0 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb456bf640 10 monclient: get_auth_request con 0x7fdb2406dd00 auth_method 0 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fdb38085010 con 0x7fdb40149dc0 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb35ffb640 10 client.4381.objecter ms_dispatch 0x7fdb40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb35ffb640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb35ffb640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb35ffb640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb47923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb47923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb47923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.074+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb401db310 con 0x7fdb40149dc0 2024-03-29T11:25:15.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.075+0000 7fdb35ffb640 10 client.4381.objecter ms_handle_connect 0x7fdb2406dd00 2024-03-29T11:25:15.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.085+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb401db310 con 0x7fdb40149dc0 2024-03-29T11:25:15.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.085+0000 7fdb35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.085+0000 7fdb35ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:15.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.410+0000 7fdb47923640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:15.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.410+0000 7fdb47923640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:15.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.410+0000 7fdb47923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:25:15.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.410+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb401db6f0 con 0x7fdb40149dc0 2024-03-29T11:25:15.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.411+0000 7fdb35ffb640 1 -- 172.21.15.138:0/1109053478 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fdb401db6f0 con 0x7fdb40149dc0 2024-03-29T11:25:15.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.411+0000 7fdb35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:15.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.411+0000 7fdb35ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:15.400 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:15.400 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:15.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 10 librados: watch_flush enter 2024-03-29T11:25:15.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 10 librados: watch_flush exit 2024-03-29T11:25:15.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T11:25:15.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 10 client.4381.objecter successfully canceled tick 2024-03-29T11:25:15.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 >> v1:172.21.15.152:6816/3052585572 conn(0x7fdb2406dd00 legacy=0x7fdb240701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 10 monclient: shutdown 2024-03-29T11:25:15.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:15.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 >> v1:172.21.15.152:6789/0 conn(0x7fdb40149dc0 legacy=0x7fdb401db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 shutdown_connections 2024-03-29T11:25:15.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 >> 172.21.15.138:0/1109053478 conn(0x7fdb400b35e0 msgr2=0x7fdb400b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:15.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.415+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 shutdown_connections 2024-03-29T11:25:15.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.416+0000 7fdb47923640 1 -- 172.21.15.138:0/1109053478 wait complete. 2024-03-29T11:25:15.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:15.416+0000 7fdb47923640 1 librados: shutdown 2024-03-29T11:25:15.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:22.222 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:22.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.394+0000 7f36c3cc1640 10 monclient: get_monmap_and_config 2024-03-29T11:25:22.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.394+0000 7f36c3cc1640 10 monclient: build_initial_monmap 2024-03-29T11:25:22.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.394+0000 7f36c3cc1640 10 monclient: monmap: 2024-03-29T11:25:22.378 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:22.395509+0000 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:22.395509+0000 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:22.379 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:22.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.394+0000 7f36c3cc1640 1 Processor -- start 2024-03-29T11:25:22.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.395+0000 7f36c3cc1640 1 -- start start 2024-03-29T11:25:22.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.395+0000 7f36c3cc1640 10 monclient: init 2024-03-29T11:25:22.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.396+0000 7f36c3cc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:22.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.396+0000 7f36c3cc1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:22.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.396+0000 7f36c3cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f36bc1457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:22.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c3cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f36bc13f650 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:22.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c3cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f36bc0a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:22.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c3cc1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0aaef0 con 0x7f36bc13f650 2024-03-29T11:25:22.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c3cc1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc145bd0 con 0x7f36bc1457f0 2024-03-29T11:25:22.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c3cc1640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc145eb0 con 0x7f36bc0a9d70 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c3cc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c3cc1640 10 monclient(hunting): authenticate will time out at 1143.708492s 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c125c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f36bc13f650 0x7f36bc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:37671/0 (socket says 172.21.15.138:37671) 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.399+0000 7f36c125c640 1 -- 172.21.15.138:0/2456010181 learned_addr learned my addr 172.21.15.138:0/2456010181 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055052444 0 0) 0x7f36bc145eb0 con 0x7f36bc0a9d70 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a8003520 con 0x7f36bc0a9d70 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748378676 0 0) 0x7f36bc0aaef0 con 0x7f36bc13f650 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:22.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc145eb0 con 0x7f36bc13f650 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056773473 0 0) 0x7f36bc145bd0 con 0x7f36bc1457f0 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc0aaef0 con 0x7f36bc1457f0 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30478241 0 0) 0x7f36a8003520 con 0x7f36bc0a9d70 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 >> v1:172.21.15.152:6789/0 conn(0x7f36bc1457f0 legacy=0x7f36bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 >> v1:172.21.15.138:6789/0 conn(0x7f36bc13f650 legacy=0x7f36bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient: _finish_hunting 0 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 monclient: found mon.noname-b 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:22.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc146340 con 0x7f36bc0a9d70 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient: _finish_auth 0 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 10 monclient: _check_auth_tickets 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.400+0000 7f36c0a5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f36b4001a10 con 0x7f36bc0a9d70 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 10 monclient: dump: 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:22.390 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b4001b70 con 0x7f36bc0a9d70 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 1 -- 172.21.15.138:0/2456010181 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f36b4001f80 con 0x7f36bc0a9d70 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c0a5b640 10 monclient: dump: 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.391 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c3cc1640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c3cc1640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.401+0000 7f36c3cc1640 4 set_mon_vals no callback set 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.402+0000 7f36c3cc1640 10 monclient: shutdown 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.402+0000 7f36c3cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.402+0000 7f36c3cc1640 1 -- 172.21.15.138:0/2456010181 >> v1:172.21.15.138:6790/0 conn(0x7f36bc0a9d70 legacy=0x7f36bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.402+0000 7f36c3cc1640 1 -- 172.21.15.138:0/2456010181 shutdown_connections 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.402+0000 7f36c3cc1640 1 -- 172.21.15.138:0/2456010181 >> 172.21.15.138:0/2456010181 conn(0x7f36bc066a00 msgr2=0x7f36bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:22.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.403+0000 7f36c3cc1640 1 -- 172.21.15.138:0/2456010181 shutdown_connections 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.403+0000 7f36c3cc1640 1 -- 172.21.15.138:0/2456010181 wait complete. 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.404+0000 7f36c3cc1640 10 monclient: build_initial_monmap 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.404+0000 7f36c3cc1640 10 monclient: monmap: 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:22.394 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.405+0000 7f36c3cc1640 1 librados: starting msgr at 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.405+0000 7f36c3cc1640 1 librados: starting objecter 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.405+0000 7f36c3cc1640 1 Processor -- start 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.405+0000 7f36c3cc1640 1 -- start start 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.405+0000 7f36c3cc1640 1 librados: setting wanted keys 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.405+0000 7f36c3cc1640 1 librados: calling monclient init 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.405+0000 7f36c3cc1640 10 monclient: init 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.406+0000 7f36c3cc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.406+0000 7f36c3cc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f36bc0a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f36bc13f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f36bc1457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc13b300 con 0x7f36bc0a9d70 2024-03-29T11:25:22.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc13b570 con 0x7f36bc1457f0 2024-03-29T11:25:22.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc13a240 con 0x7f36bc13f650 2024-03-29T11:25:22.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:22.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.407+0000 7f36c3cc1640 10 monclient(hunting): authenticate will time out at 1143.716859s 2024-03-29T11:25:22.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.408+0000 7f36c125c640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f36bc13f650 0x7f36bc136c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:47074/0 (socket says 172.21.15.138:47074) 2024-03-29T11:25:22.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.408+0000 7f36c125c640 1 -- 172.21.15.138:0/3854104028 learned_addr learned my addr 172.21.15.138:0/3854104028 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:22.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.410+0000 7f36b27fc640 10 client.?.objecter ms_handle_connect 0x7f36bc13f650 2024-03-29T11:25:22.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.410+0000 7f36b27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:22.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314790991 0 0) 0x7f36bc13a240 con 0x7f36bc13f650 2024-03-29T11:25:22.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:22.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:22.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:22.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0004eb0 con 0x7f36bc13f650 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 client.?.objecter ms_handle_connect 0x7f36bc0a9d70 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513459878 0 0) 0x7f36bc13b300 con 0x7f36bc0a9d70 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc13a240 con 0x7f36bc0a9d70 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780752237 0 0) 0x7f36a0004eb0 con 0x7f36bc13f650 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36bc13b300 con 0x7f36bc13f650 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f36ac001970 con 0x7f36bc13f650 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:22.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): dump: 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1777558014 0 0) 0x7f36bc13b300 con 0x7f36bc13f650 2024-03-29T11:25:22.408 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.413+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 >> v1:172.21.15.152:6789/0 conn(0x7f36bc1457f0 legacy=0x7f36bc1e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 >> v1:172.21.15.138:6789/0 conn(0x7f36bc0a9d70 legacy=0x7f36bc1391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 1 monclient: found mon.c 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc13a760 con 0x7f36bc13f650 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 10 monclient: _renew_subs 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc137320 con 0x7f36bc13f650 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 10 monclient: _renew_subs 2024-03-29T11:25:22.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc139bb0 con 0x7f36bc13f650 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.416+0000 7f36c3cc1640 1 librados: init done 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.417+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36ac0047e0 con 0x7f36bc13f650 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.417+0000 7f36b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.417+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f36ac004bb0 con 0x7f36bc13f650 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.417+0000 7f36b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.417+0000 7f36b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.417+0000 7f36b3fff640 4 set_mon_vals no callback set 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.417+0000 7f36b27fc640 10 monclient: dump: 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:22.410 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.418+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f36ac004dd0 con 0x7f36bc13f650 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.418+0000 7f36c3cc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.418+0000 7f36c3cc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.418+0000 7f36c3cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.418+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a4005080 con 0x7f36bc13f650 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.419+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f36ac084a50 con 0x7f36bc13f650 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.419+0000 7f36b27fc640 10 client.4445.objecter ms_dispatch 0x7f36bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.419+0000 7f36b27fc640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.419+0000 7f36b27fc640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.419+0000 7f36b27fc640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:25:22.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.420+0000 7f36c1a5d640 10 monclient: get_auth_request con 0x7f36a006db00 auth_method 0 2024-03-29T11:25:22.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.423+0000 7f36b27fc640 10 client.4445.objecter ms_handle_connect 0x7f36a006db00 2024-03-29T11:25:22.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.429+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36ac050390 con 0x7f36bc13f650 2024-03-29T11:25:22.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.429+0000 7f36b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:22.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.429+0000 7f36b27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:22.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.754+0000 7f36c3cc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.754+0000 7f36c3cc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.754+0000 7f36c3cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:22.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.754+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36a4005330 con 0x7f36bc13f650 2024-03-29T11:25:22.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.755+0000 7f36b27fc640 1 -- 172.21.15.138:0/3854104028 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f36ac084100 con 0x7f36bc13f650 2024-03-29T11:25:22.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.755+0000 7f36b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.755+0000 7f36b27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:22.739 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:22.739 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:22.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.758+0000 7f36c3cc1640 10 librados: watch_flush enter 2024-03-29T11:25:22.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.758+0000 7f36c3cc1640 10 librados: watch_flush exit 2024-03-29T11:25:22.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 >> v1:172.21.15.152:6816/3052585572 conn(0x7f36a006db00 legacy=0x7f36a006ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 10 monclient: shutdown 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 >> v1:172.21.15.138:6790/0 conn(0x7f36bc13f650 legacy=0x7f36bc136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 shutdown_connections 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 >> 172.21.15.138:0/3854104028 conn(0x7f36bc066a00 msgr2=0x7f36bc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 shutdown_connections 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 1 -- 172.21.15.138:0/3854104028 wait complete. 2024-03-29T11:25:22.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:22.759+0000 7f36c3cc1640 1 librados: shutdown 2024-03-29T11:25:22.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:29.568 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.747+0000 7ff320ab3640 10 monclient: get_monmap_and_config 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.747+0000 7ff320ab3640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.747+0000 7ff320ab3640 10 monclient: monmap: 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:29.748660+0000 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:29.748660+0000 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:29.732 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:29.733 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:29.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.748+0000 7ff320ab3640 1 Processor -- start 2024-03-29T11:25:29.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.748+0000 7ff320ab3640 1 -- start start 2024-03-29T11:25:29.733 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.748+0000 7ff320ab3640 10 monclient: init 2024-03-29T11:25:29.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:29.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 10 monclient(hunting): picked mon.noname-a con 0x7ff31c0708a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:29.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 10 monclient(hunting): picked mon.noname-c con 0x7ff31c06fe90 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:29.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 10 monclient(hunting): picked mon.noname-b con 0x7ff31c143c50 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:29.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c072630 con 0x7ff31c0708a0 2024-03-29T11:25:29.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c071410 con 0x7ff31c06fe90 2024-03-29T11:25:29.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c0afaa0 con 0x7ff31c143c50 2024-03-29T11:25:29.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.749+0000 7ff320ab3640 10 monclient(hunting): authenticate will time out at 1151.058730s 2024-03-29T11:25:29.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.750+0000 7ff31ad9d640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7ff31c143c50 0x7ff31c14b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:42186/0 (socket says 172.21.15.138:42186) 2024-03-29T11:25:29.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.750+0000 7ff31ad9d640 1 -- 172.21.15.138:0/4205915261 learned_addr learned my addr 172.21.15.138:0/4205915261 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:29.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.751+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445524038 0 0) 0x7ff31c072630 con 0x7ff31c0708a0 2024-03-29T11:25:29.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.751+0000 7ff31959a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.751+0000 7ff31959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.751+0000 7ff31959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.751+0000 7ff31959a640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:25:29.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.751+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff304003520 con 0x7ff31c0708a0 2024-03-29T11:25:29.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.752+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337631346 0 0) 0x7ff304003520 con 0x7ff31c0708a0 2024-03-29T11:25:29.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.752+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 >> v1:172.21.15.138:6790/0 conn(0x7ff31c143c50 legacy=0x7ff31c14b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:29.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.752+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 >> v1:172.21.15.152:6789/0 conn(0x7ff31c06fe90 legacy=0x7ff31c143540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:29.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 1 monclient: found mon.noname-a 2024-03-29T11:25:29.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:25:29.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c0ac8b0 con 0x7ff31c0708a0 2024-03-29T11:25:29.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff30c001690 con 0x7ff31c0708a0 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff320ab3640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff320ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: dump: 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:29.739 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:29.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff320ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff30c001920 con 0x7ff31c0708a0 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff320ab3640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff320ab3640 4 set_mon_vals no callback set 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 1 -- 172.21.15.138:0/4205915261 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff30c001cb0 con 0x7ff31c0708a0 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:29.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.754+0000 7ff31959a640 10 monclient: dump: 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.755+0000 7ff320ab3640 10 monclient: shutdown 2024-03-29T11:25:29.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.755+0000 7ff320ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:29.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.755+0000 7ff320ab3640 1 -- 172.21.15.138:0/4205915261 >> v1:172.21.15.138:6789/0 conn(0x7ff31c0708a0 legacy=0x7ff31c0af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.756+0000 7ff320ab3640 1 -- 172.21.15.138:0/4205915261 shutdown_connections 2024-03-29T11:25:29.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.756+0000 7ff320ab3640 1 -- 172.21.15.138:0/4205915261 >> 172.21.15.138:0/4205915261 conn(0x7ff31c0675b0 msgr2=0x7ff31c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:29.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.756+0000 7ff320ab3640 1 -- 172.21.15.138:0/4205915261 shutdown_connections 2024-03-29T11:25:29.747 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.756+0000 7ff320ab3640 1 -- 172.21.15.138:0/4205915261 wait complete. 2024-03-29T11:25:29.747 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 10 monclient: monmap: 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:29.748 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 1 librados: starting msgr at 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 1 librados: starting objecter 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 1 Processor -- start 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 1 -- start start 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 1 librados: setting wanted keys 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 1 librados: calling monclient init 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.757+0000 7ff320ab3640 10 monclient: init 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.758+0000 7ff320ab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.758+0000 7ff320ab3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff320ab3640 10 monclient(hunting): picked mon.noname-b con 0x7ff31c06fe90 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff320ab3640 10 monclient(hunting): picked mon.noname-a con 0x7ff31c0708a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:29.749 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff320ab3640 10 monclient(hunting): picked mon.noname-c con 0x7ff31c143c50 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff320ab3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c14b790 con 0x7ff31c0708a0 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff320ab3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c0abfa0 con 0x7ff31c06fe90 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff31a59c640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ff31c06fe90 0x7ff31c0a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:54507/0 (socket says 172.21.15.138:54507) 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff31a59c640 1 -- 172.21.15.138:0/4293777216 learned_addr learned my addr 172.21.15.138:0/4293777216 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.761+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c0ac280 con 0x7ff31c143c50 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff320ab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff320ab3640 10 monclient(hunting): authenticate will time out at 1151.071075s 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 client.?.objecter ms_handle_connect 0x7ff31c06fe90 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 client.?.objecter ms_handle_connect 0x7ff31c143c50 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 client.?.objecter ms_handle_connect 0x7ff31c0708a0 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699197248 0 0) 0x7ff31c0abfa0 con 0x7ff31c06fe90 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2fc004eb0 con 0x7ff31c06fe90 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.762+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256904159 0 0) 0x7ff31c0ac280 con 0x7ff31c143c50 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff31c0abfa0 con 0x7ff31c143c50 2024-03-29T11:25:29.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184541193 0 0) 0x7ff31c14b790 con 0x7ff31c0708a0 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff31c0ac280 con 0x7ff31c0708a0 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 182791606 0 0) 0x7ff31c0abfa0 con 0x7ff31c143c50 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff31c14b790 con 0x7ff31c143c50 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1126321469 0 0) 0x7ff2fc004eb0 con 0x7ff31c06fe90 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff31c0abfa0 con 0x7ff31c06fe90 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff314001810 con 0x7ff31c143c50 2024-03-29T11:25:29.753 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): dump: 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:29.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff30c001610 con 0x7ff31c06fe90 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.763+0000 7ff30affd640 10 monclient(hunting): dump: 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3770942923 0 0) 0x7ff31c14b790 con 0x7ff31c143c50 2024-03-29T11:25:29.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 >> v1:172.21.15.152:6789/0 conn(0x7ff31c06fe90 legacy=0x7ff31c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 >> v1:172.21.15.138:6789/0 conn(0x7ff31c0708a0 legacy=0x7ff31c140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 1 monclient: found mon.c 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c14bac0 con 0x7ff31c143c50 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff30affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 10 monclient: _renew_subs 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c14c3e0 con 0x7ff31c143c50 2024-03-29T11:25:29.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 10 monclient: _renew_subs 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c13d5c0 con 0x7ff31c143c50 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.764+0000 7ff320ab3640 1 librados: init done 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.766+0000 7ff320ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.766+0000 7ff320ab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.766+0000 7ff320ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.766+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c071e40 con 0x7ff31c143c50 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.767+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff314001d40 con 0x7ff31c143c50 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.767+0000 7ff30affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.767+0000 7ff30bfff640 4 set_mon_vals no callback set 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.768+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff314004e10 con 0x7ff31c143c50 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.768+0000 7ff30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.768+0000 7ff30affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.768+0000 7ff30affd640 10 monclient: dump: 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:29.758 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.768+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff31401ade0 con 0x7ff31c143c50 2024-03-29T11:25:29.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.769+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ff314084b80 con 0x7ff31c143c50 2024-03-29T11:25:29.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.769+0000 7ff30affd640 10 client.4448.objecter ms_dispatch 0x7ff31c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:29.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.769+0000 7ff30affd640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:29.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.769+0000 7ff30affd640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:29.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.769+0000 7ff31a59c640 10 monclient: get_auth_request con 0x7ff2fc06de90 auth_method 0 2024-03-29T11:25:29.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.769+0000 7ff30affd640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:25:29.759 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.770+0000 7ff30affd640 10 client.4448.objecter ms_handle_connect 0x7ff2fc06de90 2024-03-29T11:25:29.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.777+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3140504e0 con 0x7ff31c143c50 2024-03-29T11:25:29.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.777+0000 7ff30affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.761 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:29.777+0000 7ff30affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:30.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.146+0000 7ff320ab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.130 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.146+0000 7ff320ab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.146+0000 7ff320ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:30.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.146+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff31c14bc50 con 0x7ff31c143c50 2024-03-29T11:25:30.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.147+0000 7ff30affd640 1 -- 172.21.15.138:0/4293777216 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ff314052410 con 0x7ff31c143c50 2024-03-29T11:25:30.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.147+0000 7ff30affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.131 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.147+0000 7ff30affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:30.132 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:30.132 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:30.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 10 librados: watch_flush enter 2024-03-29T11:25:30.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 10 librados: watch_flush exit 2024-03-29T11:25:30.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:25:30.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:25:30.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff2fc06de90 legacy=0x7ff2fc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.136 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 10 monclient: shutdown 2024-03-29T11:25:30.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:30.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 >> v1:172.21.15.138:6790/0 conn(0x7ff31c143c50 legacy=0x7ff31c2000b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 shutdown_connections 2024-03-29T11:25:30.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 >> 172.21.15.138:0/4293777216 conn(0x7ff31c0675b0 msgr2=0x7ff31c06c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:30.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 shutdown_connections 2024-03-29T11:25:30.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 1 -- 172.21.15.138:0/4293777216 wait complete. 2024-03-29T11:25:30.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:30.151+0000 7ff320ab3640 1 librados: shutdown 2024-03-29T11:25:30.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:36.960 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:37.095 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.111+0000 7fb84209a640 10 monclient: get_monmap_and_config 2024-03-29T11:25:37.095 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.111+0000 7fb84209a640 10 monclient: build_initial_monmap 2024-03-29T11:25:37.095 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.112+0000 7fb84209a640 10 monclient: monmap: 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:37.112254+0000 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:37.112254+0000 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:37.096 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:37.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.113+0000 7fb84209a640 1 Processor -- start 2024-03-29T11:25:37.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.113+0000 7fb84209a640 1 -- start start 2024-03-29T11:25:37.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.113+0000 7fb84209a640 10 monclient: init 2024-03-29T11:25:37.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:37.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:37.098 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 10 monclient(hunting): picked mon.noname-c con 0x7fb83c1457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 10 monclient(hunting): picked mon.noname-b con 0x7fb83c13f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 10 monclient(hunting): picked mon.noname-a con 0x7fb83c0a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c0aaef0 con 0x7fb83c0a9d70 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c145bd0 con 0x7fb83c1457f0 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c145eb0 con 0x7fb83c13f650 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb84209a640 10 monclient(hunting): authenticate will time out at 1158.422205s 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83b7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fb83c1457f0 0x7fb83c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:55296/0 (socket says 172.21.15.138:55296) 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83b7fe640 1 -- 172.21.15.138:0/3095628185 learned_addr learned my addr 172.21.15.138:0/3095628185 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190836658 0 0) 0x7fb83c145eb0 con 0x7fb83c13f650 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:25:37.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb820003520 con 0x7fb83c13f650 2024-03-29T11:25:37.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881088721 0 0) 0x7fb820003520 con 0x7fb83c13f650 2024-03-29T11:25:37.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 >> v1:172.21.15.152:6789/0 conn(0x7fb83c1457f0 legacy=0x7fb83c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.114+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 >> v1:172.21.15.138:6789/0 conn(0x7fb83c0a9d70 legacy=0x7fb83c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c146340 con 0x7fb83c13f650 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:37.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb84209a640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb84209a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb828001690 con 0x7fb83c13f650 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb83a7fc640 10 monclient: dump: 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:37.103 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:37.104 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:37.104 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.118+0000 7fb84209a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:37.105 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb828001920 con 0x7fb83c13f650 2024-03-29T11:25:37.105 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb83a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:37.105 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb83a7fc640 1 -- 172.21.15.138:0/3095628185 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb828001d30 con 0x7fb83c13f650 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb83a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb83a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb83a7fc640 10 monclient: dump: 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:37.106 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb84209a640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.119+0000 7fb84209a640 4 set_mon_vals no callback set 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.120+0000 7fb84209a640 10 monclient: shutdown 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.120+0000 7fb84209a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.120+0000 7fb84209a640 1 -- 172.21.15.138:0/3095628185 >> v1:172.21.15.138:6790/0 conn(0x7fb83c13f650 legacy=0x7fb83c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.120+0000 7fb84209a640 1 -- 172.21.15.138:0/3095628185 shutdown_connections 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.120+0000 7fb84209a640 1 -- 172.21.15.138:0/3095628185 >> 172.21.15.138:0/3095628185 conn(0x7fb83c066a00 msgr2=0x7fb83c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.120+0000 7fb84209a640 1 -- 172.21.15.138:0/3095628185 shutdown_connections 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.121+0000 7fb84209a640 1 -- 172.21.15.138:0/3095628185 wait complete. 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.121+0000 7fb84209a640 10 monclient: build_initial_monmap 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.121+0000 7fb84209a640 10 monclient: monmap: 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:37.107 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.122+0000 7fb84209a640 1 librados: starting msgr at 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.122+0000 7fb84209a640 1 librados: starting objecter 2024-03-29T11:25:37.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.122+0000 7fb84209a640 1 Processor -- start 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.122+0000 7fb84209a640 1 -- start start 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.122+0000 7fb84209a640 1 librados: setting wanted keys 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.122+0000 7fb84209a640 1 librados: calling monclient init 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.122+0000 7fb84209a640 10 monclient: init 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.123+0000 7fb84209a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.123+0000 7fb84209a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.123+0000 7fb84209a640 10 monclient(hunting): picked mon.noname-b con 0x7fb83c0a9d70 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.123+0000 7fb83affd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fb83c13f650 0x7fb83c0b0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:49116/0 (socket says 172.21.15.138:49116) 2024-03-29T11:25:37.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.123+0000 7fb83affd640 1 -- 172.21.15.138:0/4254367140 learned_addr learned my addr 172.21.15.138:0/4254367140 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.123+0000 7fb84209a640 10 monclient(hunting): picked mon.noname-c con 0x7fb83c13f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb833fff640 10 client.?.objecter ms_handle_connect 0x7fb83c13f650 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb833fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb84209a640 10 monclient(hunting): picked mon.noname-a con 0x7fb83c1457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c0adaf0 con 0x7fb83c1457f0 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c0ad710 con 0x7fb83c0a9d70 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c0ad8a0 con 0x7fb83c13f650 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb84209a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb84209a640 10 monclient(hunting): authenticate will time out at 1158.432595s 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb833fff640 10 client.?.objecter ms_handle_connect 0x7fb83c0a9d70 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.124+0000 7fb833fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.125+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543570485 0 0) 0x7fb83c0ad8a0 con 0x7fb83c13f650 2024-03-29T11:25:37.116 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.125+0000 7fb833fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.125+0000 7fb833fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.125+0000 7fb833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.125+0000 7fb833fff640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.125+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb824004eb0 con 0x7fb83c13f650 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.125+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3289540189 0 0) 0x7fb824004eb0 con 0x7fb83c13f650 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb83c0ad8a0 con 0x7fb83c13f650 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb828001710 con 0x7fb83c13f650 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): dump: 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.117 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 client.?.objecter ms_handle_connect 0x7fb83c1457f0 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137633663 0 0) 0x7fb83c0adaf0 con 0x7fb83c1457f0 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb824004eb0 con 0x7fb83c1457f0 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 393558025 0 0) 0x7fb83c0ad8a0 con 0x7fb83c13f650 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 >> v1:172.21.15.152:6789/0 conn(0x7fb83c0a9d70 legacy=0x7fb83c1d7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 >> v1:172.21.15.138:6789/0 conn(0x7fb83c1457f0 legacy=0x7fb83c0b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.126+0000 7fb833fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb833fff640 1 monclient: found mon.c 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb833fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb833fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0ae010 con 0x7fb83c13f650 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb833fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb833fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb833fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 10 monclient: _renew_subs 2024-03-29T11:25:37.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c1d7780 con 0x7fb83c13f650 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 10 monclient: _renew_subs 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c142250 con 0x7fb83c13f650 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.127+0000 7fb84209a640 1 librados: init done 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb828003980 con 0x7fb83c13f650 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb833fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb828003dd0 con 0x7fb83c13f650 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb833fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb833fff640 10 monclient: dump: 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:37.120 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb839ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.128+0000 7fb84209a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.130+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fb828003f30 con 0x7fb83c13f650 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.130+0000 7fb84209a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.130+0000 7fb84209a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:37.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.130+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb818005080 con 0x7fb83c13f650 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.131+0000 7fb83b7fe640 10 monclient: get_auth_request con 0x7fb82406db80 auth_method 0 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.131+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fb828084c80 con 0x7fb83c13f650 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.131+0000 7fb833fff640 10 client.4454.objecter ms_dispatch 0x7fb83c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.131+0000 7fb833fff640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.131+0000 7fb833fff640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.131+0000 7fb833fff640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.131+0000 7fb833fff640 10 client.4454.objecter ms_handle_connect 0x7fb82406db80 2024-03-29T11:25:37.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.137+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb828050560 con 0x7fb83c13f650 2024-03-29T11:25:37.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.137+0000 7fb833fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.137+0000 7fb833fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:37.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.456+0000 7fb84209a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.456+0000 7fb84209a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.456+0000 7fb84209a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:37.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.456+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb818005330 con 0x7fb83c13f650 2024-03-29T11:25:37.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.459+0000 7fb833fff640 1 -- 172.21.15.138:0/4254367140 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fb828052490 con 0x7fb83c13f650 2024-03-29T11:25:37.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.459+0000 7fb833fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.459+0000 7fb833fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:37.443 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:37.443 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:37.447 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 10 librados: watch_flush enter 2024-03-29T11:25:37.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 10 librados: watch_flush exit 2024-03-29T11:25:37.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:25:37.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:25:37.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 >> v1:172.21.15.152:6816/3052585572 conn(0x7fb82406db80 legacy=0x7fb824070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 10 monclient: shutdown 2024-03-29T11:25:37.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:37.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 >> v1:172.21.15.138:6790/0 conn(0x7fb83c13f650 legacy=0x7fb83c0b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 shutdown_connections 2024-03-29T11:25:37.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.464+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 >> 172.21.15.138:0/4254367140 conn(0x7fb83c066a00 msgr2=0x7fb83c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:37.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.465+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 shutdown_connections 2024-03-29T11:25:37.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.465+0000 7fb84209a640 1 -- 172.21.15.138:0/4254367140 wait complete. 2024-03-29T11:25:37.450 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:37.466+0000 7fb84209a640 1 librados: shutdown 2024-03-29T11:25:37.466 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:44.273 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:44.428 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.443+0000 7f37f60ac640 10 monclient: get_monmap_and_config 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.443+0000 7f37f60ac640 10 monclient: build_initial_monmap 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.443+0000 7f37f60ac640 10 monclient: monmap: 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:44.443841+0000 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:44.443841+0000 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.444+0000 7f37f60ac640 1 Processor -- start 2024-03-29T11:25:44.429 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.444+0000 7f37f60ac640 1 -- start start 2024-03-29T11:25:44.430 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.444+0000 7f37f60ac640 10 monclient: init 2024-03-29T11:25:44.434 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 10 monclient(hunting): picked mon.noname-b con 0x7f37f00b3f30 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 10 monclient(hunting): picked mon.noname-c con 0x7f37f00a9750 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 10 monclient(hunting): picked mon.noname-a con 0x7f37f013b9d0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f0142490 con 0x7f37f013b9d0 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f00ac130 con 0x7f37f00a9750 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f00ac410 con 0x7f37f00b3f30 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.445+0000 7f37f60ac640 10 monclient(hunting): authenticate will time out at 1165.753774s 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.448+0000 7f37f50aa640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f37f00b3f30 0x7f37f0141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:50922/0 (socket says 172.21.15.138:50922) 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.448+0000 7f37f50aa640 1 -- 172.21.15.138:0/2825570041 learned_addr learned my addr 172.21.15.138:0/2825570041 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.449+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615105407 0 0) 0x7f37f00ac410 con 0x7f37f00b3f30 2024-03-29T11:25:44.435 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.449+0000 7f37e7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.449+0000 7f37e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.449+0000 7f37e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.449+0000 7f37e7fff640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:25:44.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.449+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37dc003520 con 0x7f37f00b3f30 2024-03-29T11:25:44.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.450+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649657213 0 0) 0x7f37dc003520 con 0x7f37f00b3f30 2024-03-29T11:25:44.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.450+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 >> v1:172.21.15.152:6789/0 conn(0x7f37f00a9750 legacy=0x7f37f013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.436 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.450+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 >> v1:172.21.15.138:6789/0 conn(0x7f37f013b9d0 legacy=0x7f37f01428f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:44.438 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 1 monclient: found mon.noname-b 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f00afa40 con 0x7f37f00b3f30 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37f60ac640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37f60ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f37ec001690 con 0x7f37f00b3f30 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37e7fff640 10 monclient: dump: 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:44.439 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.452+0000 7f37f60ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37ec001930 con 0x7f37f00b3f30 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37e7fff640 1 -- 172.21.15.138:0/2825570041 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f37ec001cc0 con 0x7f37f00b3f30 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.440 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37e7fff640 10 monclient: dump: 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37f60ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.453+0000 7f37f60ac640 4 set_mon_vals no callback set 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.454+0000 7f37f60ac640 10 monclient: shutdown 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.454+0000 7f37f60ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:44.441 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.454+0000 7f37f60ac640 1 -- 172.21.15.138:0/2825570041 >> v1:172.21.15.138:6790/0 conn(0x7f37f00b3f30 legacy=0x7f37f0141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.454+0000 7f37f60ac640 1 -- 172.21.15.138:0/2825570041 shutdown_connections 2024-03-29T11:25:44.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.454+0000 7f37f60ac640 1 -- 172.21.15.138:0/2825570041 >> 172.21.15.138:0/2825570041 conn(0x7f37f00673e0 msgr2=0x7f37f00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:44.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.456+0000 7f37f60ac640 1 -- 172.21.15.138:0/2825570041 shutdown_connections 2024-03-29T11:25:44.442 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.457+0000 7f37f60ac640 1 -- 172.21.15.138:0/2825570041 wait complete. 2024-03-29T11:25:44.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.457+0000 7f37f60ac640 10 monclient: build_initial_monmap 2024-03-29T11:25:44.443 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.457+0000 7f37f60ac640 10 monclient: monmap: 2024-03-29T11:25:44.443 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:44.443 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:44.443 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:44.443 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:44.443 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.458+0000 7f37f60ac640 1 librados: starting msgr at 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.458+0000 7f37f60ac640 1 librados: starting objecter 2024-03-29T11:25:44.444 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.458+0000 7f37f60ac640 1 Processor -- start 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.458+0000 7f37f60ac640 1 -- start start 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.458+0000 7f37f60ac640 1 librados: setting wanted keys 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.458+0000 7f37f60ac640 1 librados: calling monclient init 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.458+0000 7f37f60ac640 10 monclient: init 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 10 monclient(hunting): picked mon.noname-a con 0x7f37f00a9750 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 10 monclient(hunting): picked mon.noname-c con 0x7f37f00b3f30 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:44.448 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 10 monclient(hunting): picked mon.noname-b con 0x7f37f013b9d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f0154640 con 0x7f37f00a9750 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f013c2c0 con 0x7f37f013b9d0 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f013ef30 con 0x7f37f00b3f30 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.459+0000 7f37f60ac640 10 monclient(hunting): authenticate will time out at 1165.767460s 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.462+0000 7f37f48a9640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f37f00b3f30 0x7f37f0153820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:52906/0 (socket says 172.21.15.138:52906) 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.462+0000 7f37f48a9640 1 -- 172.21.15.138:0/2035003355 learned_addr learned my addr 172.21.15.138:0/2035003355 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.462+0000 7f37e5ffb640 10 client.?.objecter ms_handle_connect 0x7f37f00a9750 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.462+0000 7f37e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.463+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758493566 0 0) 0x7f37f0154640 con 0x7f37f00a9750 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.463+0000 7f37e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.463+0000 7f37e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.463+0000 7f37e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.449 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.463+0000 7f37e5ffb640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:25:44.450 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.463+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d4004eb0 con 0x7f37f00a9750 2024-03-29T11:25:44.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 client.?.objecter ms_handle_connect 0x7f37f00b3f30 2024-03-29T11:25:44.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 client.?.objecter ms_handle_connect 0x7f37f013b9d0 2024-03-29T11:25:44.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351053091 0 0) 0x7f37f013ef30 con 0x7f37f00b3f30 2024-03-29T11:25:44.455 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f0154640 con 0x7f37f00b3f30 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013261365 0 0) 0x7f37d4004eb0 con 0x7f37f00a9750 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f013ef30 con 0x7f37f00a9750 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f37ec0014b0 con 0x7f37f00a9750 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.456 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1777436547 0 0) 0x7f37f0154640 con 0x7f37f00b3f30 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37d4004eb0 con 0x7f37f00b3f30 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f37e8001cf0 con 0x7f37f00b3f30 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.466+0000 7f37e5ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:44.457 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1759683921 0 0) 0x7f37f013ef30 con 0x7f37f00a9750 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 >> v1:172.21.15.138:6790/0 conn(0x7f37f00b3f30 legacy=0x7f37f0153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 >> v1:172.21.15.152:6789/0 conn(0x7f37f013b9d0 legacy=0x7f37f0153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.458 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 1 monclient: found mon.a 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f013f0c0 con 0x7f37f00a9750 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 10 monclient: _renew_subs 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f013be30 con 0x7f37f00a9750 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37ec0017b0 con 0x7f37f00a9750 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:44.459 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f37ec004d90 con 0x7f37f00a9750 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37e77fe640 4 set_mon_vals no callback set 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 10 monclient: _renew_subs 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f0135470 con 0x7f37f00a9750 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.467+0000 7f37f60ac640 1 librados: init done 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.468+0000 7f37e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.468+0000 7f37e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.468+0000 7f37e5ffb640 10 monclient: dump: 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.460 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.469+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f37ec01ad60 con 0x7f37f00a9750 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37f60ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37f60ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37f60ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f013c130 con 0x7f37f00a9750 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37f48a9640 10 monclient: get_auth_request con 0x7f37d406e020 auth_method 0 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f37ec054980 con 0x7f37f00a9750 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37e5ffb640 10 client.4431.objecter ms_dispatch 0x7f37f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37e5ffb640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:44.461 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37e5ffb640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:44.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.470+0000 7f37e5ffb640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-29T11:25:44.462 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.473+0000 7f37e5ffb640 10 client.4431.objecter ms_handle_connect 0x7f37d406e020 2024-03-29T11:25:44.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.478+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37ec050490 con 0x7f37f00a9750 2024-03-29T11:25:44.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.479+0000 7f37e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.463 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.479+0000 7f37e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:44.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.810+0000 7f37f60ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.810+0000 7f37f60ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.810+0000 7f37f60ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:25:44.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.810+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37f013c500 con 0x7f37f00a9750 2024-03-29T11:25:44.797 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.813+0000 7f37e5ffb640 1 -- 172.21.15.138:0/2035003355 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f37ec0523c0 con 0x7f37f00a9750 2024-03-29T11:25:44.797 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.813+0000 7f37e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.797 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.813+0000 7f37e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:44.797 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:44.797 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:44.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 10 librados: watch_flush enter 2024-03-29T11:25:44.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 10 librados: watch_flush exit 2024-03-29T11:25:44.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-29T11:25:44.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 10 client.4431.objecter successfully canceled tick 2024-03-29T11:25:44.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 >> v1:172.21.15.152:6816/3052585572 conn(0x7f37d406e020 legacy=0x7f37d40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 10 monclient: shutdown 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 >> v1:172.21.15.138:6789/0 conn(0x7f37f00a9750 legacy=0x7f37f013c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 shutdown_connections 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.817+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 >> 172.21.15.138:0/2035003355 conn(0x7f37f00673e0 msgr2=0x7f37f0128d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.818+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 shutdown_connections 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.818+0000 7f37f60ac640 1 -- 172.21.15.138:0/2035003355 wait complete. 2024-03-29T11:25:44.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:44.818+0000 7f37f60ac640 1 librados: shutdown 2024-03-29T11:25:44.818 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:51.625 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:51.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.792+0000 7f6dd878f640 10 monclient: get_monmap_and_config 2024-03-29T11:25:51.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.792+0000 7f6dd878f640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.792+0000 7f6dd878f640 10 monclient: monmap: 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:51.793547+0000 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:51.793547+0000 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.794+0000 7f6dd878f640 1 Processor -- start 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.794+0000 7f6dd878f640 1 -- start start 2024-03-29T11:25:51.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.794+0000 7f6dd878f640 10 monclient: init 2024-03-29T11:25:51.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:51.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 10 monclient(hunting): picked mon.noname-b con 0x7f6dd00a9750 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:51.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 10 monclient(hunting): picked mon.noname-a con 0x7f6dd0149ac0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 10 monclient(hunting): picked mon.noname-c con 0x7f6dd014fe50 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd00aa4a0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd014a680 con 0x7f6dd014fe50 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd00af290 con 0x7f6dd00a9750 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd878f640 10 monclient(hunting): authenticate will time out at 1173.105266s 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd652b640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f6dd00a9750 0x7f6dd01436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:41864/0 (socket says 172.21.15.138:41864) 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd652b640 1 -- 172.21.15.138:0/3685909980 learned_addr learned my addr 172.21.15.138:0/3685909980 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd652b640 1 -- 172.21.15.138:0/3685909980 >> v1:172.21.15.138:6790/0 conn(0x7f6dd00a9750 legacy=0x7f6dd01436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd652b640 1 -- 172.21.15.138:0/3685909980 >> v1:172.21.15.138:6790/0 conn(0x7f6dd00a9750 legacy=0x7f6dd01436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:25:51.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.796+0000 7f6dd652b640 1 --1- 172.21.15.138:0/3685909980 >> v1:172.21.15.138:6790/0 conn(0x7f6dd00a9750 0x7f6dd01436f0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971217988 0 0) 0x7f6dd00aa4a0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db8003520 con 0x7f6dd0149ac0 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2929044167 0 0) 0x7f6db8003520 con 0x7f6dd0149ac0 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 >> v1:172.21.15.138:6790/0 conn(0x7f6dd00a9750 legacy=0x7f6dd01436f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 >> v1:172.21.15.152:6789/0 conn(0x7f6dd014fe50 legacy=0x7f6dd0152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 monclient: found mon.noname-a 2024-03-29T11:25:51.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd00afb40 con 0x7f6dd0149ac0 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6dc0001690 con 0x7f6dd0149ac0 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: dump: 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.792 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd878f640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd878f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc0001860 con 0x7f6dd0149ac0 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 1 -- 172.21.15.138:0/3685909980 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6dc0001cb0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd5529640 10 monclient: dump: 2024-03-29T11:25:51.793 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd878f640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.803+0000 7f6dd878f640 4 set_mon_vals no callback set 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.804+0000 7f6dd878f640 10 monclient: shutdown 2024-03-29T11:25:51.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.804+0000 7f6dd878f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:51.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.804+0000 7f6dd878f640 1 -- 172.21.15.138:0/3685909980 >> v1:172.21.15.138:6789/0 conn(0x7f6dd0149ac0 legacy=0x7f6dd0149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.804+0000 7f6dd878f640 1 -- 172.21.15.138:0/3685909980 shutdown_connections 2024-03-29T11:25:51.795 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.805+0000 7f6dd878f640 1 -- 172.21.15.138:0/3685909980 >> 172.21.15.138:0/3685909980 conn(0x7f6dd00b35e0 msgr2=0x7f6dd00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.807+0000 7f6dd878f640 1 -- 172.21.15.138:0/3685909980 shutdown_connections 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.808+0000 7f6dd878f640 1 -- 172.21.15.138:0/3685909980 wait complete. 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.808+0000 7f6dd878f640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.808+0000 7f6dd878f640 10 monclient: monmap: 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:51.796 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.797 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:51.797 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:51.797 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:51.797 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:51.797 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:51.797 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.809+0000 7f6dd878f640 1 librados: starting msgr at 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.809+0000 7f6dd878f640 1 librados: starting objecter 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.809+0000 7f6dd878f640 1 Processor -- start 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.811+0000 7f6dd878f640 1 -- start start 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.811+0000 7f6dd878f640 1 librados: setting wanted keys 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.811+0000 7f6dd878f640 1 librados: calling monclient init 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.811+0000 7f6dd878f640 10 monclient: init 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.799 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 10 monclient(hunting): picked mon.noname-a con 0x7f6dd00a9750 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 10 monclient(hunting): picked mon.noname-c con 0x7f6dd0149ac0 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 10 monclient(hunting): picked mon.noname-b con 0x7f6dd014fe50 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd013e070 con 0x7f6dd00a9750 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd0148eb0 con 0x7f6dd014fe50 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd0148b40 con 0x7f6dd0149ac0 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.812+0000 7f6dd878f640 10 monclient(hunting): authenticate will time out at 1173.121226s 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.814+0000 7f6dd5d2a640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f6dd0149ac0 0x7f6dd01db860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:52630/0 (socket says 172.21.15.138:52630) 2024-03-29T11:25:51.800 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.814+0000 7f6dd5d2a640 1 -- 172.21.15.138:0/4193014682 learned_addr learned my addr 172.21.15.138:0/4193014682 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:51.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 client.?.objecter ms_handle_connect 0x7f6dd0149ac0 2024-03-29T11:25:51.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138593879 0 0) 0x7f6dd0148b40 con 0x7f6dd0149ac0 2024-03-29T11:25:51.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.801 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc004eb0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892883182 0 0) 0x7f6dbc004eb0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dd0148b40 con 0x7f6dd0149ac0 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 client.?.objecter ms_handle_connect 0x7f6dd014fe50 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6dc0001bb0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): dump: 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:51.802 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.815+0000 7f6dceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3334690934 0 0) 0x7f6dd0148b40 con 0x7f6dd0149ac0 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 >> v1:172.21.15.152:6789/0 conn(0x7f6dd014fe50 legacy=0x7f6dd01dbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.803 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 >> v1:172.21.15.138:6789/0 conn(0x7f6dd00a9750 legacy=0x7f6dd00aca40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:51.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 1 monclient: found mon.c 2024-03-29T11:25:51.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:51.805 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd01dc680 con 0x7f6dd0149ac0 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.816+0000 7f6dceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 10 monclient: _renew_subs 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd00ad150 con 0x7f6dd0149ac0 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 10 monclient: _renew_subs 2024-03-29T11:25:51.806 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd00ac0f0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd878f640 1 librados: init done 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc0003940 con 0x7f6dd0149ac0 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.817+0000 7f6dd4d28640 4 set_mon_vals no callback set 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.818+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6dc0003d50 con 0x7f6dd0149ac0 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.818+0000 7f6dceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.818+0000 7f6dceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.818+0000 7f6dceffd640 10 monclient: dump: 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.807 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.819+0000 7f6dd878f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.819+0000 7f6dd878f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.819+0000 7f6dd878f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:51.808 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.819+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd0148520 con 0x7f6dd0149ac0 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.821+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f6dc0003eb0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.822+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f6dc0084bf0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.822+0000 7f6dceffd640 10 client.4463.objecter ms_dispatch 0x7f6dd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.822+0000 7f6dceffd640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.822+0000 7f6dceffd640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.822+0000 7f6dceffd640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.822+0000 7f6dd652b640 10 monclient: get_auth_request con 0x7f6dbc06dc90 auth_method 0 2024-03-29T11:25:51.810 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.823+0000 7f6dceffd640 10 client.4463.objecter ms_handle_connect 0x7f6dbc06dc90 2024-03-29T11:25:51.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.829+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6dc00504d0 con 0x7f6dd0149ac0 2024-03-29T11:25:51.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.829+0000 7f6dceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.814 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:51.829+0000 7f6dceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:52.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.161+0000 7f6dd878f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:52.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.162+0000 7f6dd878f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:52.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.162+0000 7f6dd878f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:52.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.162+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dd0148cd0 con 0x7f6dd0149ac0 2024-03-29T11:25:52.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.165+0000 7f6dceffd640 1 -- 172.21.15.138:0/4193014682 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f6dc0052400 con 0x7f6dd0149ac0 2024-03-29T11:25:52.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.165+0000 7f6dceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:52.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.165+0000 7f6dceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:52.150 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:52.150 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:52.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 10 librados: watch_flush enter 2024-03-29T11:25:52.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 10 librados: watch_flush exit 2024-03-29T11:25:52.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:25:52.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:25:52.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 >> v1:172.21.15.152:6816/3052585572 conn(0x7f6dbc06dc90 legacy=0x7f6dbc070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 10 monclient: shutdown 2024-03-29T11:25:52.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:52.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 >> v1:172.21.15.138:6790/0 conn(0x7f6dd0149ac0 legacy=0x7f6dd01db860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 shutdown_connections 2024-03-29T11:25:52.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.169+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 >> 172.21.15.138:0/4193014682 conn(0x7f6dd00b35e0 msgr2=0x7f6dd00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:52.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.170+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 shutdown_connections 2024-03-29T11:25:52.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.172+0000 7f6dd878f640 1 -- 172.21.15.138:0/4193014682 wait complete. 2024-03-29T11:25:52.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:52.172+0000 7f6dd878f640 1 librados: shutdown 2024-03-29T11:25:52.177 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:58.985 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:59.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.151+0000 7f22aee33640 10 monclient: get_monmap_and_config 2024-03-29T11:25:59.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.151+0000 7f22aee33640 10 monclient: build_initial_monmap 2024-03-29T11:25:59.137 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.151+0000 7f22aee33640 10 monclient: monmap: 2024-03-29T11:25:59.137 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:59.137 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:59.137 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:25:59.152475+0000 2024-03-29T11:25:59.138 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:25:59.152475+0000 2024-03-29T11:25:59.138 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:59.138 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:59.138 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:59.138 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:25:59.138 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:25:59.138 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.153+0000 7f22aee33640 1 Processor -- start 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.154+0000 7f22aee33640 1 -- start start 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.154+0000 7f22aee33640 10 monclient: init 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.154+0000 7f22aee33640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.154+0000 7f22aee33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.154+0000 7f22aee33640 10 monclient(hunting): picked mon.noname-a con 0x7f22a80b0860 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.155+0000 7f22aee33640 10 monclient(hunting): picked mon.noname-b con 0x7f22a813ef30 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.155+0000 7f22aee33640 10 monclient(hunting): picked mon.noname-c con 0x7f22a80af8e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:59.142 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.155+0000 7f22aee33640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80b0c40 con 0x7f22a80b0860 2024-03-29T11:25:59.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.155+0000 7f22aee33640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80aed10 con 0x7f22a80af8e0 2024-03-29T11:25:59.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.155+0000 7f22aee33640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80afd50 con 0x7f22a813ef30 2024-03-29T11:25:59.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.155+0000 7f22aee33640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:59.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.155+0000 7f22aee33640 10 monclient(hunting): authenticate will time out at 1180.464116s 2024-03-29T11:25:59.143 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.158+0000 7f22a7fff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f22a813ef30 0x7f22a80af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:57470/0 (socket says 172.21.15.138:57470) 2024-03-29T11:25:59.144 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.158+0000 7f22a7fff640 1 -- 172.21.15.138:0/216106221 learned_addr learned my addr 172.21.15.138:0/216106221 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:59.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349654339 0 0) 0x7f22a80afd50 con 0x7f22a813ef30 2024-03-29T11:25:59.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:59.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2290003520 con 0x7f22a813ef30 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067961746 0 0) 0x7f2290003520 con 0x7f22a813ef30 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 >> v1:172.21.15.152:6789/0 conn(0x7f22a80af8e0 legacy=0x7f22a8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 >> v1:172.21.15.138:6789/0 conn(0x7f22a80b0860 legacy=0x7f22a813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:25:59.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80ab1b0 con 0x7f22a813ef30 2024-03-29T11:25:59.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:59.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:59.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:59.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f229c001690 con 0x7f22a813ef30 2024-03-29T11:25:59.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.159+0000 7f22a77fe640 10 monclient: dump: 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:59.150 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22aee33640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22aee33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f229c001860 con 0x7f22a813ef30 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22a77fe640 1 -- 172.21.15.138:0/216106221 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f229c001cb0 con 0x7f22a813ef30 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22a77fe640 10 monclient: dump: 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:59.151 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22aee33640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.160+0000 7f22aee33640 4 set_mon_vals no callback set 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.161+0000 7f22aee33640 10 monclient: shutdown 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.161+0000 7f22aee33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:59.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.161+0000 7f22aee33640 1 -- 172.21.15.138:0/216106221 >> v1:172.21.15.138:6790/0 conn(0x7f22a813ef30 legacy=0x7f22a80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.161+0000 7f22aee33640 1 -- 172.21.15.138:0/216106221 shutdown_connections 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.161+0000 7f22aee33640 1 -- 172.21.15.138:0/216106221 >> 172.21.15.138:0/216106221 conn(0x7f22a806b660 msgr2=0x7f22a806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.162+0000 7f22aee33640 1 -- 172.21.15.138:0/216106221 shutdown_connections 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.162+0000 7f22aee33640 1 -- 172.21.15.138:0/216106221 wait complete. 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.163+0000 7f22aee33640 10 monclient: build_initial_monmap 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.163+0000 7f22aee33640 10 monclient: monmap: 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:59.153 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:25:59.154 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:25:59.154 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:25:59.154 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:59.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.163+0000 7f22aee33640 1 librados: starting msgr at 2024-03-29T11:25:59.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.163+0000 7f22aee33640 1 librados: starting objecter 2024-03-29T11:25:59.154 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.163+0000 7f22aee33640 1 Processor -- start 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.164+0000 7f22aee33640 1 -- start start 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.164+0000 7f22aee33640 1 librados: setting wanted keys 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.164+0000 7f22aee33640 1 librados: calling monclient init 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.164+0000 7f22aee33640 10 monclient: init 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.165+0000 7f22aee33640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.165+0000 7f22aee33640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 10 monclient(hunting): picked mon.noname-a con 0x7f22a80af8e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:25:59.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 10 monclient(hunting): picked mon.noname-b con 0x7f22a80b0860 addr v1:172.21.15.152:6789/0 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 10 monclient(hunting): picked mon.noname-c con 0x7f22a813ef30 addr v1:172.21.15.138:6790/0 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80b2610 con 0x7f22a80af8e0 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a81e3d00 con 0x7f22a80b0860 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a81e40a0 con 0x7f22a813ef30 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.166+0000 7f22aee33640 10 monclient(hunting): authenticate will time out at 1180.475603s 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.167+0000 7f22acbcf640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f22a80af8e0 0x7f22a81e45c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:46071/0 (socket says 172.21.15.138:46071) 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.167+0000 7f22acbcf640 1 -- 172.21.15.138:0/2537173131 learned_addr learned my addr 172.21.15.138:0/2537173131 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 client.?.objecter ms_handle_connect 0x7f22a80af8e0 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233549244 0 0) 0x7f22a80b2610 con 0x7f22a80af8e0 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:59.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2294004eb0 con 0x7f22a80af8e0 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 client.?.objecter ms_handle_connect 0x7f22a813ef30 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.168+0000 7f22a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.169+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3946455298 0 0) 0x7f2294004eb0 con 0x7f22a80af8e0 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.169+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22a80b2610 con 0x7f22a80af8e0 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.169+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f22a0001ce0 con 0x7f22a80af8e0 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.169+0000 7f22a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.169+0000 7f22a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.169+0000 7f22a57fa640 10 monclient(hunting): dump: 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:59.162 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.169+0000 7f22a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.170+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802760117 0 0) 0x7f22a81e40a0 con 0x7f22a813ef30 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.170+0000 7f22a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.170+0000 7f22a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.170+0000 7f22a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.170+0000 7f22a57fa640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.170+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2294004eb0 con 0x7f22a813ef30 2024-03-29T11:25:59.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 10 client.?.objecter ms_handle_connect 0x7f22a80b0860 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1120710442 0 0) 0x7f2294004eb0 con 0x7f22a813ef30 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22a81e40a0 con 0x7f22a813ef30 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2298001760 con 0x7f22a813ef30 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.171+0000 7f22a57fa640 10 monclient(hunting): dump: 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.164 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1399100833 0 0) 0x7f22a81e40a0 con 0x7f22a813ef30 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 >> v1:172.21.15.152:6789/0 conn(0x7f22a80b0860 legacy=0x7f22a81e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 >> v1:172.21.15.138:6789/0 conn(0x7f22a80af8e0 legacy=0x7f22a81e45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 1 monclient: found mon.c 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a81e4cd0 con 0x7f22a813ef30 2024-03-29T11:25:59.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 10 monclient: _renew_subs 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a8138540 con 0x7f22a813ef30 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 10 monclient: _renew_subs 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a8137c10 con 0x7f22a813ef30 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22aee33640 1 librados: init done 2024-03-29T11:25:59.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2298001c30 con 0x7f22a813ef30 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f2298003790 con 0x7f22a813ef30 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a57fa640 10 monclient: dump: 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:25:59.167 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:25:59.168 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.172+0000 7f22a6ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:59.168 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.174+0000 7f22aee33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:59.168 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.174+0000 7f22aee33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:59.168 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.174+0000 7f22aee33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:59.168 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.174+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a80ab770 con 0x7f22a813ef30 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.177+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f22980038f0 con 0x7f22a813ef30 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.178+0000 7f22acbcf640 10 monclient: get_auth_request con 0x7f229406df80 auth_method 0 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.178+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f22980836e0 con 0x7f22a813ef30 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.178+0000 7f22a57fa640 10 client.4469.objecter ms_dispatch 0x7f22a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.178+0000 7f22a57fa640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.178+0000 7f22a57fa640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.178+0000 7f22a57fa640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:25:59.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.178+0000 7f22a57fa640 10 client.4469.objecter ms_handle_connect 0x7f229406df80 2024-03-29T11:25:59.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.186+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22a80ab770 con 0x7f22a813ef30 2024-03-29T11:25:59.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.186+0000 7f22a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:59.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.186+0000 7f22a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:59.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.512+0000 7f22aee33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.512+0000 7f22aee33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.512+0000 7f22aee33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:25:59.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.512+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22a80ab770 con 0x7f22a813ef30 2024-03-29T11:25:59.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.514+0000 7f22a57fa640 1 -- 172.21.15.138:0/2537173131 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f22a80ab770 con 0x7f22a813ef30 2024-03-29T11:25:59.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.514+0000 7f22a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.514+0000 7f22a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:59.500 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:25:59.500 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:59.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 10 librados: watch_flush enter 2024-03-29T11:25:59.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 10 librados: watch_flush exit 2024-03-29T11:25:59.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:25:59.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:25:59.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 >> v1:172.21.15.152:6816/3052585572 conn(0x7f229406df80 legacy=0x7f2294070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 10 monclient: shutdown 2024-03-29T11:25:59.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:59.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.518+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 >> v1:172.21.15.138:6790/0 conn(0x7f22a813ef30 legacy=0x7f22a81e35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.520+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 shutdown_connections 2024-03-29T11:25:59.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.520+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 >> 172.21.15.138:0/2537173131 conn(0x7f22a806b660 msgr2=0x7f22a80ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:59.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.521+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 shutdown_connections 2024-03-29T11:25:59.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.521+0000 7f22aee33640 1 -- 172.21.15.138:0/2537173131 wait complete. 2024-03-29T11:25:59.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:25:59.521+0000 7f22aee33640 1 librados: shutdown 2024-03-29T11:25:59.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:06.334 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:06.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.501+0000 7f96c5990640 10 monclient: get_monmap_and_config 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.501+0000 7f96c5990640 10 monclient: build_initial_monmap 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.501+0000 7f96c5990640 10 monclient: monmap: 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:06.502419+0000 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:06.502419+0000 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:06.487 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:06.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.503+0000 7f96c5990640 1 Processor -- start 2024-03-29T11:26:06.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.503+0000 7f96c5990640 1 -- start start 2024-03-29T11:26:06.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.503+0000 7f96c5990640 10 monclient: init 2024-03-29T11:26:06.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.504+0000 7f96c5990640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:06.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.504+0000 7f96c5990640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:06.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.504+0000 7f96c5990640 10 monclient(hunting): picked mon.noname-c con 0x7f96b80389e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96c5990640 10 monclient(hunting): picked mon.noname-a con 0x7f96b8038dc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96c5990640 10 monclient(hunting): picked mon.noname-b con 0x7f96b8030660 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96c5990640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b80c54c0 con 0x7f96b8038dc0 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96c5990640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b8036430 con 0x7f96b80389e0 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96c5990640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b8039840 con 0x7f96b8030660 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96c5990640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96c5990640 10 monclient(hunting): authenticate will time out at 1187.814573s 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96bf7fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f96b8030660 0x7f96b80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:37550/0 (socket says 172.21.15.138:37550) 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.505+0000 7f96bf7fe640 1 -- 172.21.15.138:0/2930516788 learned_addr learned my addr 172.21.15.138:0/2930516788 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.506+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017816634 0 0) 0x7f96b8039840 con 0x7f96b8030660 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.506+0000 7f96bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.493 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.506+0000 7f96bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.506+0000 7f96bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.506+0000 7f96bdffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:26:06.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.506+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a8003520 con 0x7f96b8030660 2024-03-29T11:26:06.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.507+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 551228096 0 0) 0x7f96a8003520 con 0x7f96b8030660 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.507+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 >> v1:172.21.15.152:6789/0 conn(0x7f96b80389e0 legacy=0x7f96b8033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 >> v1:172.21.15.138:6789/0 conn(0x7f96b8038dc0 legacy=0x7f96b80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 1 monclient: found mon.noname-b 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b8035690 con 0x7f96b8030660 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96c004f730 con 0x7f96b8030660 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.508+0000 7f96bdffb640 10 monclient: dump: 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96c5990640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96c5990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:06.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96c0075f10 con 0x7f96b8030660 2024-03-29T11:26:06.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96bdffb640 1 -- 172.21.15.138:0/2930516788 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96c00758b0 con 0x7f96b8030660 2024-03-29T11:26:06.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96bdffb640 10 monclient: dump: 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:06.498 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96c5990640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.509+0000 7f96c5990640 4 set_mon_vals no callback set 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.510+0000 7f96c5990640 10 monclient: shutdown 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.510+0000 7f96c5990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.510+0000 7f96c5990640 1 -- 172.21.15.138:0/2930516788 >> v1:172.21.15.138:6790/0 conn(0x7f96b8030660 legacy=0x7f96b80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.510+0000 7f96c5990640 1 -- 172.21.15.138:0/2930516788 shutdown_connections 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.510+0000 7f96c5990640 1 -- 172.21.15.138:0/2930516788 >> 172.21.15.138:0/2930516788 conn(0x7f96b8005c00 msgr2=0x7f96b8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.511+0000 7f96c5990640 1 -- 172.21.15.138:0/2930516788 shutdown_connections 2024-03-29T11:26:06.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.512+0000 7f96c5990640 1 -- 172.21.15.138:0/2930516788 wait complete. 2024-03-29T11:26:06.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.512+0000 7f96c5990640 10 monclient: build_initial_monmap 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.512+0000 7f96c5990640 10 monclient: monmap: 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.513+0000 7f96c5990640 1 librados: starting msgr at 2024-03-29T11:26:06.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.513+0000 7f96c5990640 1 librados: starting objecter 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.513+0000 7f96c5990640 1 Processor -- start 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.513+0000 7f96c5990640 1 -- start start 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.513+0000 7f96c5990640 1 librados: setting wanted keys 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.513+0000 7f96c5990640 1 librados: calling monclient init 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.513+0000 7f96c5990640 10 monclient: init 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.514+0000 7f96c5990640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.514+0000 7f96c5990640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.514+0000 7f96c5990640 10 monclient(hunting): picked mon.noname-c con 0x7f96b8030660 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.515+0000 7f96c5990640 10 monclient(hunting): picked mon.noname-a con 0x7f96b80389e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.515+0000 7f96c5990640 10 monclient(hunting): picked mon.noname-b con 0x7f96b80c2ad0 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.515+0000 7f96c5990640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b802ec60 con 0x7f96b80389e0 2024-03-29T11:26:06.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.515+0000 7f96c5990640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b802ef40 con 0x7f96b80c2ad0 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.515+0000 7f96c5990640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b802fe30 con 0x7f96b8030660 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.515+0000 7f96c5990640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.515+0000 7f96c5990640 10 monclient(hunting): authenticate will time out at 1187.824646s 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96beffd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f96b8030660 0x7f96b80c23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:40142/0 (socket says 172.21.15.138:40142) 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96beffd640 1 -- 172.21.15.138:0/2611811223 learned_addr learned my addr 172.21.15.138:0/2611811223 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 10 client.?.objecter ms_handle_connect 0x7f96b8030660 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764945804 0 0) 0x7f96b802fe30 con 0x7f96b8030660 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.516+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac004eb0 con 0x7f96b8030660 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.517+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172742618 0 0) 0x7f96ac004eb0 con 0x7f96b8030660 2024-03-29T11:26:06.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.517+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96b802fe30 con 0x7f96b8030660 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.517+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96b4001870 con 0x7f96b8030660 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.517+0000 7f96a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.517+0000 7f96a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.517+0000 7f96a7fff640 10 monclient(hunting): dump: 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.509 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.517+0000 7f96a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109078175 0 0) 0x7f96b802fe30 con 0x7f96b8030660 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 >> v1:172.21.15.152:6789/0 conn(0x7f96b80c2ad0 legacy=0x7f96b8168520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 >> v1:172.21.15.138:6789/0 conn(0x7f96b80389e0 legacy=0x7f96b8160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 1 monclient: found mon.c 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b802e940 con 0x7f96b8030660 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.518+0000 7f96a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 10 monclient: _renew_subs 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 1 -- 172.21.15.138:0/2611811223 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b80c7b60 con 0x7f96b8030660 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 10 monclient: _renew_subs 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 1 -- 172.21.15.138:0/2611811223 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b802fa40 con 0x7f96b8030660 2024-03-29T11:26:06.511 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96c5990640 1 librados: init done 2024-03-29T11:26:06.513 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b4001ad0 con 0x7f96b8030660 2024-03-29T11:26:06.513 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.513 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.519+0000 7f96bcff9640 4 set_mon_vals no callback set 2024-03-29T11:26:06.513 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.520+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f96b4004b60 con 0x7f96b8030660 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.520+0000 7f96a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.520+0000 7f96a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.520+0000 7f96a7fff640 10 monclient: dump: 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:06.514 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.522+0000 7f96a5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.525+0000 7f96a5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.525+0000 7f96a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.525+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b8007b90 con 0x7f96b8030660 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.525+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f96b4004cc0 con 0x7f96b8030660 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.526+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f96b4084be0 con 0x7f96b8030660 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.526+0000 7f96a7fff640 10 client.4475.objecter ms_dispatch 0x7f96c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.526+0000 7f96a7fff640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.526+0000 7f96a7fff640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.526+0000 7f96a7fff640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:26:06.515 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.527+0000 7f96be7fc640 10 monclient: get_auth_request con 0x7f96ac06dc10 auth_method 0 2024-03-29T11:26:06.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.530+0000 7f96a7fff640 10 client.4475.objecter ms_handle_connect 0x7f96ac06dc10 2024-03-29T11:26:06.522 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.537+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96b40502f0 con 0x7f96b8030660 2024-03-29T11:26:06.523 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.537+0000 7f96a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.523 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.537+0000 7f96a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:06.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.871+0000 7f96a5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.871+0000 7f96a5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.871+0000 7f96a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:06.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.871+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96b802f3b0 con 0x7f96b8030660 2024-03-29T11:26:06.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.874+0000 7f96a7fff640 1 -- 172.21.15.138:0/2611811223 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f96b802f3b0 con 0x7f96b8030660 2024-03-29T11:26:06.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.874+0000 7f96a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.874+0000 7f96a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:06.860 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:06.860 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:06.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.877+0000 7f96a5ffb640 10 librados: watch_flush enter 2024-03-29T11:26:06.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.877+0000 7f96a5ffb640 10 librados: watch_flush exit 2024-03-29T11:26:06.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.877+0000 7f96a5ffb640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:26:06.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.877+0000 7f96a5ffb640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:26:06.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.878+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 >> v1:172.21.15.152:6816/3052585572 conn(0x7f96ac06dc10 legacy=0x7f96ac0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.878+0000 7f96a5ffb640 10 monclient: shutdown 2024-03-29T11:26:06.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.878+0000 7f96a5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.878+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 >> v1:172.21.15.138:6790/0 conn(0x7f96b8030660 legacy=0x7f96b80c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.878+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 shutdown_connections 2024-03-29T11:26:06.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.878+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 >> 172.21.15.138:0/2611811223 conn(0x7f96b8005c00 msgr2=0x7f96b8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.879+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 shutdown_connections 2024-03-29T11:26:06.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.879+0000 7f96a5ffb640 1 -- 172.21.15.138:0/2611811223 wait complete. 2024-03-29T11:26:06.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:06.880+0000 7f96a5ffb640 1 librados: shutdown 2024-03-29T11:26:06.886 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:13.694 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:13.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.868+0000 7ff8949d6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:13.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.868+0000 7ff8949d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.868+0000 7ff8949d6640 10 monclient: monmap: 2024-03-29T11:26:13.854 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:13.854 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:13.869578+0000 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:13.869578+0000 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:13.855 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:13.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.869+0000 7ff8949d6640 1 Processor -- start 2024-03-29T11:26:13.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.869+0000 7ff8949d6640 1 -- start start 2024-03-29T11:26:13.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.869+0000 7ff8949d6640 10 monclient: init 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8901457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff89013f650 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8900a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900aaef0 con 0x7ff89013f650 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff890145bd0 con 0x7ff8901457f0 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff890145eb0 con 0x7ff8900a9d70 2024-03-29T11:26:13.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.870+0000 7ff8949d6640 10 monclient(hunting): authenticate will time out at 1195.179742s 2024-03-29T11:26:13.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.873+0000 7ff88e59c640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ff8901457f0 0x7ff89013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:45352/0 (socket says 172.21.15.138:45352) 2024-03-29T11:26:13.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.873+0000 7ff88e59c640 1 -- 172.21.15.138:0/3695226617 learned_addr learned my addr 172.21.15.138:0/3695226617 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:13.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330302776 0 0) 0x7ff890145bd0 con 0x7ff8901457f0 2024-03-29T11:26:13.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff878003520 con 0x7ff8901457f0 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224056417 0 0) 0x7ff890145eb0 con 0x7ff8900a9d70 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890145bd0 con 0x7ff8900a9d70 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2995179405 0 0) 0x7ff878003520 con 0x7ff8901457f0 2024-03-29T11:26:13.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.874+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.138:6790/0 conn(0x7ff8900a9d70 legacy=0x7ff89014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.876+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.138:6789/0 conn(0x7ff89013f650 legacy=0x7ff8900b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 1 monclient: found mon.noname-c 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890146340 con 0x7ff8901457f0 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.877+0000 7ff88d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff8949d6640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff8949d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:13.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff880001690 con 0x7ff8901457f0 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient: dump: 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:13.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff8949d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:13.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88e59c640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.152:6789/0 conn(0x7ff8901457f0 legacy=0x7ff89013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:26:13.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88e59c640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.152:6789/0 conn(0x7ff8901457f0 legacy=0x7ff89013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:26:13.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88e59c640 1 --1- 172.21.15.138:0/3695226617 >> v1:172.21.15.152:6789/0 conn(0x7ff8901457f0 0x7ff89013ef40 :-1 s=OPENED pgs=739504 cs=1 l=1).handle_message read tag failed 2024-03-29T11:26:13.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88e59c640 1 --1- 172.21.15.138:0/3695226617 >> v1:172.21.15.152:6789/0 conn(0x7ff8901457f0 0x7ff89013ef40 :-1 s=OPENED pgs=739504 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:26:13.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient: ms_handle_reset current mon v1:172.21.15.152:6789/0 2024-03-29T11:26:13.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.152:6789/0 conn(0x7ff8901457f0 legacy=0x7ff89013ef40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:26:13.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:13.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient(hunting): picked mon.c con 0x7ff8900a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:13.867 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.878+0000 7ff88d59a640 10 monclient(hunting): picked mon.a con 0x7ff89013f650 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:13.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.882+0000 7ff88d59a640 10 monclient(hunting): picked mon.b con 0x7ff87800b090 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:13.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.882+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff878003520 con 0x7ff89013f650 2024-03-29T11:26:13.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.882+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff8900aaef0 con 0x7ff87800b090 2024-03-29T11:26:13.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.882+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff890145eb0 con 0x7ff8900a9d70 2024-03-29T11:26:13.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.882+0000 7ff88d59a640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.882+0000 7ff88d59a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.152:6789/0 2024-03-29T11:26:13.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726832836 0 0) 0x7ff8900aaef0 con 0x7ff87800b090 2024-03-29T11:26:13.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.152:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7ff87800ee00 con 0x7ff87800b090 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704150382 0 0) 0x7ff878003520 con 0x7ff89013f650 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.138:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7ff8900aaef0 con 0x7ff89013f650 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1468401994 0 0) 0x7ff87800ee00 con 0x7ff87800b090 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.138:6790/0 conn(0x7ff8900a9d70 legacy=0x7ff878003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.138:6789/0 conn(0x7ff89013f650 legacy=0x7ff878009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 monclient: found mon.b 2024-03-29T11:26:13.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7ff878012090 con 0x7ff87800b090 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff888001b20 con 0x7ff87800b090 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: dump: 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.871 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff8949d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 1 -- 172.21.15.138:0/3695226617 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff888001d40 con 0x7ff87800b090 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.883+0000 7ff88d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.884+0000 7ff8949d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:13.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.884+0000 7ff8949d6640 4 set_mon_vals no callback set 2024-03-29T11:26:13.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.885+0000 7ff8949d6640 10 monclient: shutdown 2024-03-29T11:26:13.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.885+0000 7ff8949d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:13.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.885+0000 7ff8949d6640 1 -- 172.21.15.138:0/3695226617 >> v1:172.21.15.152:6789/0 conn(0x7ff87800b090 legacy=0x7ff87800d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.885+0000 7ff8949d6640 1 -- 172.21.15.138:0/3695226617 shutdown_connections 2024-03-29T11:26:13.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.885+0000 7ff8949d6640 1 -- 172.21.15.138:0/3695226617 >> 172.21.15.138:0/3695226617 conn(0x7ff890066a00 msgr2=0x7ff890066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:13.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.886+0000 7ff8949d6640 1 -- 172.21.15.138:0/3695226617 shutdown_connections 2024-03-29T11:26:13.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.886+0000 7ff8949d6640 1 -- 172.21.15.138:0/3695226617 wait complete. 2024-03-29T11:26:13.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.887+0000 7ff8949d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.887+0000 7ff8949d6640 10 monclient: monmap: 2024-03-29T11:26:13.874 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:13.874 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:13.874 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.888+0000 7ff8949d6640 1 librados: starting msgr at 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.888+0000 7ff8949d6640 1 librados: starting objecter 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.888+0000 7ff8949d6640 1 Processor -- start 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.888+0000 7ff8949d6640 1 -- start start 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.888+0000 7ff8949d6640 1 librados: setting wanted keys 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.888+0000 7ff8949d6640 1 librados: calling monclient init 2024-03-29T11:26:13.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.888+0000 7ff8949d6640 10 monclient: init 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff87800b090 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8900a9d70 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff89013f650 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8901429c0 con 0x7ff89013f650 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89013c9d0 con 0x7ff8900a9d70 2024-03-29T11:26:13.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8901df800 con 0x7ff87800b090 2024-03-29T11:26:13.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.889+0000 7ff8949d6640 10 monclient(hunting): authenticate will time out at 1195.198600s 2024-03-29T11:26:13.878 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.892+0000 7ff88dd9b640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ff8900a9d70 0x7ff89013ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:46390/0 (socket says 172.21.15.138:46390) 2024-03-29T11:26:13.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.892+0000 7ff88dd9b640 1 -- 172.21.15.138:0/3123909632 learned_addr learned my addr 172.21.15.138:0/3123909632 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:13.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 client.?.objecter ms_handle_connect 0x7ff8900a9d70 2024-03-29T11:26:13.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 client.?.objecter ms_handle_connect 0x7ff87800b090 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031980628 0 0) 0x7ff89013c9d0 con 0x7ff8900a9d70 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.893+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870004eb0 con 0x7ff8900a9d70 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 client.?.objecter ms_handle_connect 0x7ff89013f650 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581494108 0 0) 0x7ff8901df800 con 0x7ff87800b090 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:26:13.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89013c9d0 con 0x7ff87800b090 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3062435050 0 0) 0x7ff870004eb0 con 0x7ff8900a9d70 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8901df800 con 0x7ff8900a9d70 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff884001980 con 0x7ff8900a9d70 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): dump: 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.882 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.883 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:13.883 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:13.883 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:13.883 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:13.883 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:13.883 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.894+0000 7ff87effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030719660 0 0) 0x7ff8901429c0 con 0x7ff89013f650 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870004eb0 con 0x7ff89013f650 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3097008575 0 0) 0x7ff89013c9d0 con 0x7ff87800b090 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8901429c0 con 0x7ff87800b090 2024-03-29T11:26:13.885 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3930849206 0 0) 0x7ff8901df800 con 0x7ff8900a9d70 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 >> v1:172.21.15.138:6790/0 conn(0x7ff87800b090 legacy=0x7ff8901df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 >> v1:172.21.15.138:6789/0 conn(0x7ff89013f650 legacy=0x7ff89013d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 monclient: found mon.b 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8901df990 con 0x7ff8900a9d70 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.886 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff87effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff8949d6640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff8949d6640 10 monclient: _renew_subs 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff8949d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8901dfc40 con 0x7ff8900a9d70 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff8949d6640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.898+0000 7ff8949d6640 10 monclient: _renew_subs 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff8949d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:26:13.887 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8901e7620 con 0x7ff8900a9d70 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff8949d6640 1 librados: init done 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8840035f0 con 0x7ff8900a9d70 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff87effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff884003980 con 0x7ff8900a9d70 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff87effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff87effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff88cd99640 4 set_mon_vals no callback set 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.899+0000 7ff87effd640 10 monclient: dump: 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.888 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:13.889 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:13.889 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:13.889 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:13.889 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:13.890 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.901+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff884003ae0 con 0x7ff8900a9d70 2024-03-29T11:26:13.890 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.900+0000 7ff8949d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.890 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff8949d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.890 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff8949d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff874005080 con 0x7ff8900a9d70 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ff884084990 con 0x7ff8900a9d70 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff87effd640 10 client.4411.objecter ms_dispatch 0x7ff890000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff87effd640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff87effd640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff87effd640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.904+0000 7ff88e59c640 10 monclient: get_auth_request con 0x7ff87006df60 auth_method 0 2024-03-29T11:26:13.891 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.905+0000 7ff87effd640 10 client.4411.objecter ms_handle_connect 0x7ff87006df60 2024-03-29T11:26:13.895 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.909+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff884050270 con 0x7ff8900a9d70 2024-03-29T11:26:13.895 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.909+0000 7ff87effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.896 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:13.909+0000 7ff87effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:14.252 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.266+0000 7ff8949d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:14.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.266+0000 7ff8949d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:14.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.266+0000 7ff8949d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:26:14.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.266+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff874005330 con 0x7ff8900a9d70 2024-03-29T11:26:14.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.268+0000 7ff87effd640 1 -- 172.21.15.138:0/3123909632 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ff8840521a0 con 0x7ff8900a9d70 2024-03-29T11:26:14.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.268+0000 7ff87effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:14.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.268+0000 7ff87effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:14.255 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:14.255 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:14.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.272+0000 7ff8949d6640 10 librados: watch_flush enter 2024-03-29T11:26:14.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.273+0000 7ff8949d6640 10 librados: watch_flush exit 2024-03-29T11:26:14.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.273+0000 7ff8949d6640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:26:14.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.273+0000 7ff8949d6640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:26:14.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.274+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff87006df60 legacy=0x7ff870070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.274+0000 7ff8949d6640 10 monclient: shutdown 2024-03-29T11:26:14.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.274+0000 7ff8949d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:14.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.274+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 >> v1:172.21.15.152:6789/0 conn(0x7ff8900a9d70 legacy=0x7ff89013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.277+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 shutdown_connections 2024-03-29T11:26:14.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.277+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 >> 172.21.15.138:0/3123909632 conn(0x7ff890066a00 msgr2=0x7ff890136da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:14.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.280+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 shutdown_connections 2024-03-29T11:26:14.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.280+0000 7ff8949d6640 1 -- 172.21.15.138:0/3123909632 wait complete. 2024-03-29T11:26:14.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:14.282+0000 7ff8949d6640 1 librados: shutdown 2024-03-29T11:26:14.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:21.098 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.264+0000 7fc6092ae640 10 monclient: get_monmap_and_config 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.264+0000 7fc6092ae640 10 monclient: build_initial_monmap 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.264+0000 7fc6092ae640 10 monclient: monmap: 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:21.265607+0000 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:21.265607+0000 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:21.251 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:21.252 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:21.252 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:21.252 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:21.252 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:21.252 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.265+0000 7fc6092ae640 1 Processor -- start 2024-03-29T11:26:21.252 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.265+0000 7fc6092ae640 1 -- start start 2024-03-29T11:26:21.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.265+0000 7fc6092ae640 10 monclient: init 2024-03-29T11:26:21.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:21.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:21.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc6040ae250 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:21.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc6040a9720 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:21.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc60414bba0 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:21.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040a9b80 con 0x7fc6040a9720 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040aa7b0 con 0x7fc6040ae250 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040afbf0 con 0x7fc60414bba0 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6092ae640 10 monclient(hunting): authenticate will time out at 1202.575565s 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6027fc640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc6040a9720 0x7fc60413f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:39014/0 (socket says 172.21.15.138:39014) 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.266+0000 7fc6027fc640 1 -- 172.21.15.138:0/2590819309 learned_addr learned my addr 172.21.15.138:0/2590819309 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955966029 0 0) 0x7fc6040a9b80 con 0x7fc6040a9720 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:21.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:26:21.255 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f0003520 con 0x7fc6040a9720 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3194885520 0 0) 0x7fc5f0003520 con 0x7fc6040a9720 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.267+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 >> v1:172.21.15.138:6790/0 conn(0x7fc60414bba0 legacy=0x7fc60414df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 >> v1:172.21.15.152:6789/0 conn(0x7fc6040ae250 legacy=0x7fc604142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 1 monclient: found mon.noname-a 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc604143280 con 0x7fc6040a9720 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:21.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc6092ae640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc6092ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc5f4001690 con 0x7fc6040a9720 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 10 monclient: dump: 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.263 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc6092ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.268+0000 7fc601ffb640 1 -- 172.21.15.138:0/2590819309 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5f40018b0 con 0x7fc6040a9720 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.269+0000 7fc601ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.269+0000 7fc6092ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.269+0000 7fc6092ae640 4 set_mon_vals no callback set 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.269+0000 7fc6092ae640 10 monclient: shutdown 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.269+0000 7fc6092ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:21.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.269+0000 7fc6092ae640 1 -- 172.21.15.138:0/2590819309 >> v1:172.21.15.138:6789/0 conn(0x7fc6040a9720 legacy=0x7fc60413f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.270+0000 7fc6092ae640 1 -- 172.21.15.138:0/2590819309 shutdown_connections 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.270+0000 7fc6092ae640 1 -- 172.21.15.138:0/2590819309 >> 172.21.15.138:0/2590819309 conn(0x7fc6040b35e0 msgr2=0x7fc6040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.270+0000 7fc6092ae640 1 -- 172.21.15.138:0/2590819309 shutdown_connections 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.270+0000 7fc6092ae640 1 -- 172.21.15.138:0/2590819309 wait complete. 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.271+0000 7fc6092ae640 10 monclient: build_initial_monmap 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.271+0000 7fc6092ae640 10 monclient: monmap: 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:21.265 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.271+0000 7fc6092ae640 1 librados: starting msgr at 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.271+0000 7fc6092ae640 1 librados: starting objecter 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.272+0000 7fc6092ae640 1 Processor -- start 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.272+0000 7fc6092ae640 1 -- start start 2024-03-29T11:26:21.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.272+0000 7fc6092ae640 1 librados: setting wanted keys 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.272+0000 7fc6092ae640 1 librados: calling monclient init 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.272+0000 7fc6092ae640 10 monclient: init 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc6092ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc6092ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc6092ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc6040a9720 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc6092ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc6040ae250 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc6092ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc60414bba0 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc6092ae640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60413af90 con 0x7fc60414bba0 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc602ffd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fc6040a9720 0x7fc6041daf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:55780/0 (socket says 172.21.15.138:55780) 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc602ffd640 1 -- 172.21.15.138:0/3037400463 learned_addr learned my addr 172.21.15.138:0/3037400463 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:21.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.273+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6041daa70 con 0x7fc6040ae250 2024-03-29T11:26:21.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc6040a9720 2024-03-29T11:26:21.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:21.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc6040ae250 2024-03-29T11:26:21.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:21.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc60414bba0 2024-03-29T11:26:21.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:21.268 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60413b200 con 0x7fc6040a9720 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc6092ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc6092ae640 10 monclient(hunting): authenticate will time out at 1202.583507s 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310301981 0 0) 0x7fc6041daa70 con 0x7fc6040ae250 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e4004ee0 con 0x7fc6040ae250 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493281915 0 0) 0x7fc60413b200 con 0x7fc6040a9720 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:21.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6041daa70 con 0x7fc6040a9720 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472566418 0 0) 0x7fc60413af90 con 0x7fc60414bba0 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.274+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e4005390 con 0x7fc60414bba0 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3419972623 0 0) 0x7fc5e4004ee0 con 0x7fc6040ae250 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc60413af90 con 0x7fc6040ae250 2024-03-29T11:26:21.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc5f4000fe0 con 0x7fc6040ae250 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:21.271 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452787531 0 0) 0x7fc6041daa70 con 0x7fc6040a9720 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5e4004ee0 con 0x7fc6040a9720 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc5fc004340 con 0x7fc6040a9720 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.272 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3052334067 0 0) 0x7fc5e4005390 con 0x7fc60414bba0 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6041daa70 con 0x7fc60414bba0 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc5ec001750 con 0x7fc60414bba0 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:21.273 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3587682701 0 0) 0x7fc5e4004ee0 con 0x7fc6040a9720 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 >> v1:172.21.15.152:6789/0 conn(0x7fc6040ae250 legacy=0x7fc60413bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 >> v1:172.21.15.138:6789/0 conn(0x7fc60414bba0 legacy=0x7fc6041e2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 monclient: found mon.c 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60413f9f0 con 0x7fc6040a9720 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc5fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 10 monclient: _renew_subs 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6041db660 con 0x7fc6040a9720 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:21.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 10 monclient: _renew_subs 2024-03-29T11:26:21.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:21.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60413f2a0 con 0x7fc6040a9720 2024-03-29T11:26:21.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.275+0000 7fc6092ae640 1 librados: init done 2024-03-29T11:26:21.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.277+0000 7fc6092ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:21.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.277+0000 7fc6092ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:21.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.277+0000 7fc6092ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:21.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.277+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6041db850 con 0x7fc6040a9720 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.277+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5fc001250 con 0x7fc6040a9720 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.278+0000 7fc5fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.278+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc5fc0016a0 con 0x7fc6040a9720 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.278+0000 7fc5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.278+0000 7fc5fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.278+0000 7fc5fb7fe640 10 monclient: dump: 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.279 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.278+0000 7fc6017fa640 4 set_mon_vals no callback set 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.281+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fc5fc001880 con 0x7fc6040a9720 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.282+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fc5fc084960 con 0x7fc6040a9720 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.282+0000 7fc5fb7fe640 10 client.4487.objecter ms_dispatch 0x7fc604000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.282+0000 7fc5fb7fe640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.282+0000 7fc5fb7fe640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:21.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.282+0000 7fc5fb7fe640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:26:21.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.282+0000 7fc6027fc640 10 monclient: get_auth_request con 0x7fc5e406e450 auth_method 0 2024-03-29T11:26:21.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.283+0000 7fc5fb7fe640 10 client.4487.objecter ms_handle_connect 0x7fc5e406e450 2024-03-29T11:26:21.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.293+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5fc050380 con 0x7fc6040a9720 2024-03-29T11:26:21.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.293+0000 7fc5fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:21.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.293+0000 7fc5fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:21.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.669+0000 7fc6092ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.669+0000 7fc6092ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.669+0000 7fc6092ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:21.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.669+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc604075f10 con 0x7fc6040a9720 2024-03-29T11:26:21.657 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:21.657 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:21.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.670+0000 7fc5fb7fe640 1 -- 172.21.15.138:0/3037400463 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fc5fc0522b0 con 0x7fc6040a9720 2024-03-29T11:26:21.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.670+0000 7fc5fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.670+0000 7fc5fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:21.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 10 librados: watch_flush enter 2024-03-29T11:26:21.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 10 librados: watch_flush exit 2024-03-29T11:26:21.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:26:21.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:26:21.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 >> v1:172.21.15.152:6816/3052585572 conn(0x7fc5e406e450 legacy=0x7fc5e4070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 10 monclient: shutdown 2024-03-29T11:26:21.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:21.661 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.674+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 >> v1:172.21.15.138:6790/0 conn(0x7fc6040a9720 legacy=0x7fc6041daf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.675+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 shutdown_connections 2024-03-29T11:26:21.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.675+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 >> 172.21.15.138:0/3037400463 conn(0x7fc6040b35e0 msgr2=0x7fc604069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:21.662 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.675+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 shutdown_connections 2024-03-29T11:26:21.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.676+0000 7fc6092ae640 1 -- 172.21.15.138:0/3037400463 wait complete. 2024-03-29T11:26:21.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:21.676+0000 7fc6092ae640 1 librados: shutdown 2024-03-29T11:26:21.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:28.487 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:28.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.639+0000 7f4a2860d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:28.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.639+0000 7f4a2860d640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.639+0000 7f4a2860d640 10 monclient: monmap: 2024-03-29T11:26:28.626 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:28.626 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:28.640210+0000 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:28.640210+0000 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:28.627 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:28.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.643+0000 7f4a2860d640 1 Processor -- start 2024-03-29T11:26:28.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.643+0000 7f4a2860d640 1 -- start start 2024-03-29T11:26:28.629 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.643+0000 7f4a2860d640 10 monclient: init 2024-03-29T11:26:28.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:28.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a200ae250 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:28.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a2013f970 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:28.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a20142800 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:28.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200afe00 con 0x7f4a2013f970 2024-03-29T11:26:28.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a20142c10 con 0x7f4a20142800 2024-03-29T11:26:28.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200afb20 con 0x7f4a200ae250 2024-03-29T11:26:28.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.644+0000 7f4a2860d640 10 monclient(hunting): authenticate will time out at 1209.952548s 2024-03-29T11:26:28.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a263a9640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f4a200ae250 0x7f4a2013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:51496/0 (socket says 172.21.15.138:51496) 2024-03-29T11:26:28.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a263a9640 1 -- 172.21.15.138:0/4187327220 learned_addr learned my addr 172.21.15.138:0/4187327220 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:28.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911951144 0 0) 0x7f4a200afb20 con 0x7f4a200ae250 2024-03-29T11:26:28.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:26:28.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0c003520 con 0x7f4a200ae250 2024-03-29T11:26:28.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745227012 0 0) 0x7f4a0c003520 con 0x7f4a200ae250 2024-03-29T11:26:28.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.645+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 >> v1:172.21.15.152:6789/0 conn(0x7f4a20142800 legacy=0x7f4a2014da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:28.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 >> v1:172.21.15.138:6789/0 conn(0x7f4a2013f970 legacy=0x7f4a201420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:28.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 1 monclient: found mon.noname-b 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a200aab80 con 0x7f4a200ae250 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a2860d640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4a10001690 con 0x7f4a200ae250 2024-03-29T11:26:28.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a2860d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a253a7640 10 monclient: dump: 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:28.640 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.651+0000 7f4a2860d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a10001860 con 0x7f4a200ae250 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a253a7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a2860d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a2860d640 4 set_mon_vals no callback set 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a253a7640 1 -- 172.21.15.138:0/4187327220 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4a10001cb0 con 0x7f4a200ae250 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a253a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a253a7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a253a7640 10 monclient: dump: 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.641 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.642 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:28.642 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:28.642 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:28.642 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:28.642 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:28.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.653+0000 7f4a2860d640 10 monclient: shutdown 2024-03-29T11:26:28.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.654+0000 7f4a2860d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.654+0000 7f4a2860d640 1 -- 172.21.15.138:0/4187327220 >> v1:172.21.15.138:6790/0 conn(0x7f4a200ae250 legacy=0x7f4a2013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.654+0000 7f4a2860d640 1 -- 172.21.15.138:0/4187327220 shutdown_connections 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.654+0000 7f4a2860d640 1 -- 172.21.15.138:0/4187327220 >> 172.21.15.138:0/4187327220 conn(0x7f4a200b35e0 msgr2=0x7f4a200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.654+0000 7f4a2860d640 1 -- 172.21.15.138:0/4187327220 shutdown_connections 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.654+0000 7f4a2860d640 1 -- 172.21.15.138:0/4187327220 wait complete. 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 10 monclient: monmap: 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:28.644 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 1 librados: starting msgr at 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 1 librados: starting objecter 2024-03-29T11:26:28.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 1 Processor -- start 2024-03-29T11:26:28.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 1 -- start start 2024-03-29T11:26:28.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 1 librados: setting wanted keys 2024-03-29T11:26:28.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 1 librados: calling monclient init 2024-03-29T11:26:28.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.655+0000 7f4a2860d640 10 monclient: init 2024-03-29T11:26:28.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:28.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a200ae250 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a2013f970 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a20142800 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a20144d40 con 0x7f4a2013f970 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2006e7a0 con 0x7f4a200ae250 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2006f2d0 con 0x7f4a20142800 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.656+0000 7f4a2860d640 10 monclient(hunting): authenticate will time out at 1209.964748s 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.660+0000 7f4a263a9640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f4a200ae250 0x7f4a20144370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:54779/0 (socket says 172.21.15.138:54779) 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.660+0000 7f4a263a9640 1 -- 172.21.15.138:0/4015908137 learned_addr learned my addr 172.21.15.138:0/4015908137 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.660+0000 7f4a26baa640 1 --1- 172.21.15.138:0/4015908137 >> v1:172.21.15.138:6790/0 conn(0x7f4a20142800 0x7f4a201eeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:54676/0 (socket says 172.21.15.138:54676) 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 client.?.objecter ms_handle_connect 0x7f4a20142800 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 client.?.objecter ms_handle_connect 0x7f4a200ae250 2024-03-29T11:26:28.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925399083 0 0) 0x7f4a2006f2d0 con 0x7f4a20142800 2024-03-29T11:26:28.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:26:28.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a04004eb0 con 0x7f4a20142800 2024-03-29T11:26:28.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3043069937 0 0) 0x7f4a04004eb0 con 0x7f4a20142800 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a2006f2d0 con 0x7f4a20142800 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4a1c001750 con 0x7f4a20142800 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:28.649 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:28.650 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:28.650 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 247831137 0 0) 0x7f4a2006f2d0 con 0x7f4a20142800 2024-03-29T11:26:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 >> v1:172.21.15.152:6789/0 conn(0x7f4a200ae250 legacy=0x7f4a20144370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.650 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.661+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 >> v1:172.21.15.138:6789/0 conn(0x7f4a2013f970 legacy=0x7f4a2006ebc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:28.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 1 monclient: found mon.c 2024-03-29T11:26:28.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2013c2a0 con 0x7f4a20142800 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a2860d640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a2860d640 10 monclient: _renew_subs 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a2860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.666+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2013c550 con 0x7f4a20142800 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a2860d640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a2860d640 10 monclient: _renew_subs 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a2860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a201453b0 con 0x7f4a20142800 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a2860d640 1 librados: init done 2024-03-29T11:26:28.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a1c001ce0 con 0x7f4a20142800 2024-03-29T11:26:28.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a24ba6640 4 set_mon_vals no callback set 2024-03-29T11:26:28.654 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4a1c004940 con 0x7f4a20142800 2024-03-29T11:26:28.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.667+0000 7f4a16ffd640 10 monclient: dump: 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:28.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.668+0000 7f4a2860d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.668+0000 7f4a2860d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.668+0000 7f4a2860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.668+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a08005080 con 0x7f4a20142800 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.669+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f4a1c01a910 con 0x7f4a20142800 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.669+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f4a1c084980 con 0x7f4a20142800 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.669+0000 7f4a16ffd640 10 client.4493.objecter ms_dispatch 0x7f4a20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.669+0000 7f4a16ffd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.669+0000 7f4a16ffd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.669+0000 7f4a16ffd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.670+0000 7f4a263a9640 10 monclient: get_auth_request con 0x7f4a0406dbd0 auth_method 0 2024-03-29T11:26:28.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.670+0000 7f4a16ffd640 10 client.4493.objecter ms_handle_connect 0x7f4a0406dbd0 2024-03-29T11:26:28.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.676+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a1c050090 con 0x7f4a20142800 2024-03-29T11:26:28.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.676+0000 7f4a16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.663 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.676+0000 7f4a16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:28.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.968+0000 7f4a2860d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.968+0000 7f4a2860d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.968+0000 7f4a2860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:28.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.968+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a08005330 con 0x7f4a20142800 2024-03-29T11:26:28.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.969+0000 7f4a16ffd640 1 -- 172.21.15.138:0/4015908137 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f4a1c052190 con 0x7f4a20142800 2024-03-29T11:26:28.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.969+0000 7f4a16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.969+0000 7f4a16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:28.955 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:28.955 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 10 librados: watch_flush enter 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 10 librados: watch_flush exit 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 >> v1:172.21.15.152:6816/3052585572 conn(0x7f4a0406dbd0 legacy=0x7f4a04070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 10 monclient: shutdown 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 >> v1:172.21.15.138:6790/0 conn(0x7f4a20142800 legacy=0x7f4a201eeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 shutdown_connections 2024-03-29T11:26:28.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 >> 172.21.15.138:0/4015908137 conn(0x7f4a200b35e0 msgr2=0x7f4a200a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 shutdown_connections 2024-03-29T11:26:28.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 1 -- 172.21.15.138:0/4015908137 wait complete. 2024-03-29T11:26:28.959 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:28.972+0000 7f4a2860d640 1 librados: shutdown 2024-03-29T11:26:28.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:35.781 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:35.919 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.933+0000 7fa2c1d6c640 10 monclient: get_monmap_and_config 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.933+0000 7fa2c1d6c640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.933+0000 7fa2c1d6c640 10 monclient: monmap: 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:35.933966+0000 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:35.933966+0000 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.934+0000 7fa2c1d6c640 1 Processor -- start 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.934+0000 7fa2c1d6c640 1 -- start start 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.934+0000 7fa2c1d6c640 10 monclient: init 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.935+0000 7fa2c1d6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.935+0000 7fa2c1d6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2c1d6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc0b0860 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2c1d6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa2bc13ef30 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2c1d6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc0af8e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2bbfff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fa2bc0af8e0 0x7fa2bc149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50816/0 (socket says 172.21.15.138:50816) 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2bbfff640 1 -- 172.21.15.138:0/533085878 learned_addr learned my addr 172.21.15.138:0/533085878 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0b0c40 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0aed10 con 0x7fa2bc13ef30 2024-03-29T11:26:35.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.936+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0afd50 con 0x7fa2bc0b0860 2024-03-29T11:26:35.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2c1d6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2c1d6c640 10 monclient(hunting): authenticate will time out at 1217.245149s 2024-03-29T11:26:35.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076232872 0 0) 0x7fa2bc0b0c40 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2baffd640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.937+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c003520 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.938+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 52717574 0 0) 0x7fa29c003520 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.938+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 >> v1:172.21.15.138:6790/0 conn(0x7fa2bc0b0860 legacy=0x7fa2bc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.938+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 >> v1:172.21.15.152:6789/0 conn(0x7fa2bc13ef30 legacy=0x7fa2bc0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 1 monclient: found mon.noname-a 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0ab1b0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa2b40019f0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:35.941 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: dump: 2024-03-29T11:26:35.941 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:35.942 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:35.942 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.942 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.942 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.942 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:35.942 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:35.942 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:35.943 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:35.943 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:35.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2c1d6c640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:26:35.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2c1d6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:35.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2b4001c80 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 1 -- 172.21.15.138:0/533085878 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa2b4002050 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.939+0000 7fa2baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:35.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.940+0000 7fa2baffd640 10 monclient: dump: 2024-03-29T11:26:35.944 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:35.944 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:35.945 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.945 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.945 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.945 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:35.945 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:35.945 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.940+0000 7fa2c1d6c640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.940+0000 7fa2c1d6c640 4 set_mon_vals no callback set 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.940+0000 7fa2c1d6c640 10 monclient: shutdown 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.941+0000 7fa2c1d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.941+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 >> v1:172.21.15.138:6789/0 conn(0x7fa2bc0af8e0 legacy=0x7fa2bc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.941+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 shutdown_connections 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.941+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 >> 172.21.15.138:0/533085878 conn(0x7fa2bc06b660 msgr2=0x7fa2bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:35.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.941+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 shutdown_connections 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.942+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/533085878 wait complete. 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.943+0000 7fa2c1d6c640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.943+0000 7fa2c1d6c640 10 monclient: monmap: 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.943+0000 7fa2c1d6c640 1 librados: starting msgr at 2024-03-29T11:26:35.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.943+0000 7fa2c1d6c640 1 librados: starting objecter 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.943+0000 7fa2c1d6c640 1 Processor -- start 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.943+0000 7fa2c1d6c640 1 -- start start 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.943+0000 7fa2c1d6c640 1 librados: setting wanted keys 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.944+0000 7fa2c1d6c640 1 librados: calling monclient init 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.944+0000 7fa2c1d6c640 10 monclient: init 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.944+0000 7fa2c1d6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2c1d6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2c1d6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc0af8e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2bb7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fa2bc0af8e0 0x7fa2bc1d6d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:52420/0 (socket says 172.21.15.138:52420) 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2bb7fe640 1 -- 172.21.15.138:0/675544503 learned_addr learned my addr 172.21.15.138:0/675544503 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2c1d6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc0b0860 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa2bc0af8e0 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2c1d6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa2bc13ef30 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:35.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa2bc13ef30 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.945+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc1d6800 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc1d7430 con 0x7fa2bc0b0860 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc1d75c0 con 0x7fa2bc13ef30 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727027264 0 0) 0x7fa2bc1d6800 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2c1d6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2c1d6c640 10 monclient(hunting): authenticate will time out at 1217.254506s 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2b8ff9640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a0004eb0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.946+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651537991 0 0) 0x7fa2bc1d75c0 con 0x7fa2bc13ef30 2024-03-29T11:26:35.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2bc1d6800 con 0x7fa2bc13ef30 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222903583 0 0) 0x7fa2a0004eb0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2bc1d75c0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa2a40019d0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.950 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 176118615 0 0) 0x7fa2bc1d6800 con 0x7fa2bc13ef30 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2a0004eb0 con 0x7fa2bc13ef30 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa2b4002050 con 0x7fa2bc13ef30 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:35.951 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1749558294 0 0) 0x7fa2bc1d75c0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 >> v1:172.21.15.138:6790/0 conn(0x7fa2bc13ef30 legacy=0x7fa2bc132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.947+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 >> v1:172.21.15.152:6789/0 conn(0x7fa2bc0b0860 legacy=0x7fa2bc0b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 1 monclient: found mon.a 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0b2f80 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa2bc0b0860 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 10 monclient: _renew_subs 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc0b1c60 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 10 monclient: _renew_subs 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:35.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc13a2c0 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2c1d6c640 1 librados: init done 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 10 client.4452.objecter resend_mon_ops 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2a4001b30 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.948+0000 7fa2b9ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.949+0000 7fa2c1d6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa2a4003a60 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2b8ff9640 10 monclient: dump: 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:35.954 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2c1d6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2c1d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa294005080 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.950+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fa2a4003c80 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.951+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fa2a4084a10 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.951+0000 7fa2b8ff9640 10 client.4452.objecter ms_dispatch 0x7fa2bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:35.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.951+0000 7fa2b8ff9640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:35.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.951+0000 7fa2b8ff9640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:35.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.951+0000 7fa2b8ff9640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-29T11:26:35.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.954+0000 7fa2b2ffd640 10 monclient: get_auth_request con 0x7fa2a006de30 auth_method 0 2024-03-29T11:26:35.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.954+0000 7fa2b8ff9640 10 client.4452.objecter ms_handle_connect 0x7fa2a006de30 2024-03-29T11:26:35.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.958+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2a4050430 con 0x7fa2bc0af8e0 2024-03-29T11:26:35.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.959+0000 7fa2b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:35.959+0000 7fa2b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:36.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.280+0000 7fa2c1d6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:36.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.280+0000 7fa2c1d6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:36.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.280+0000 7fa2c1d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:36.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.280+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa294005330 con 0x7fa2bc0af8e0 2024-03-29T11:26:36.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.283+0000 7fa2b8ff9640 1 -- 172.21.15.138:0/675544503 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fa2a40840c0 con 0x7fa2bc0af8e0 2024-03-29T11:26:36.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.283+0000 7fa2b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:36.269 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.283+0000 7fa2b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:36.269 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:36.269 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:36.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.287+0000 7fa2c1d6c640 10 librados: watch_flush enter 2024-03-29T11:26:36.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.287+0000 7fa2c1d6c640 10 librados: watch_flush exit 2024-03-29T11:26:36.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.287+0000 7fa2c1d6c640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-29T11:26:36.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.287+0000 7fa2c1d6c640 10 client.4452.objecter successfully canceled tick 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 >> v1:172.21.15.152:6816/3052585572 conn(0x7fa2a006de30 legacy=0x7fa2a00702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 10 monclient: shutdown 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 >> v1:172.21.15.138:6789/0 conn(0x7fa2bc0af8e0 legacy=0x7fa2bc1d6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 shutdown_connections 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 >> 172.21.15.138:0/675544503 conn(0x7fa2bc06b660 msgr2=0x7fa2bc13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 shutdown_connections 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 1 -- 172.21.15.138:0/675544503 wait complete. 2024-03-29T11:26:36.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:36.288+0000 7fa2c1d6c640 1 librados: shutdown 2024-03-29T11:26:36.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:43.099 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:43.249 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.262+0000 7f457f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:43.249 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.262+0000 7f457f59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:43.249 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.262+0000 7f457f59e640 10 monclient: monmap: 2024-03-29T11:26:43.249 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:43.249 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:43.249 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:43.263466+0000 2024-03-29T11:26:43.249 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:43.263466+0000 2024-03-29T11:26:43.250 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:43.250 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:43.250 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:43.250 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:43.250 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:43.250 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:43.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.263+0000 7f457f59e640 1 Processor -- start 2024-03-29T11:26:43.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.263+0000 7f457f59e640 1 -- start start 2024-03-29T11:26:43.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.263+0000 7f457f59e640 10 monclient: init 2024-03-29T11:26:43.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.264+0000 7f457f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:43.251 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.264+0000 7f457f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:43.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.264+0000 7f457f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f45780389e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:43.253 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f4578038dc0 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f4578030660 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457f59e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45780c54c0 con 0x7f4578030660 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457f59e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4578036430 con 0x7f45780389e0 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457f59e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4578039840 con 0x7f4578038dc0 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457f59e640 10 monclient(hunting): authenticate will time out at 1224.575629s 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457dd9b640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f4578038dc0 0x7f45780cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:46950/0 (socket says 172.21.15.138:46950) 2024-03-29T11:26:43.254 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.266+0000 7f457dd9b640 1 -- 172.21.15.138:0/2194278158 learned_addr learned my addr 172.21.15.138:0/2194278158 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:43.256 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724557830 0 0) 0x7f4578039840 con 0x7f4578038dc0 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4568003520 con 0x7f4578038dc0 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784500027 0 0) 0x7f4568003520 con 0x7f4578038dc0 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 >> v1:172.21.15.152:6789/0 conn(0x7f45780389e0 legacy=0x7f4578033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 >> v1:172.21.15.138:6789/0 conn(0x7f4578030660 legacy=0x7f45780d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 monclient: found mon.noname-b 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4578035690 con 0x7f4578038dc0 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: _finish_auth 0 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4574001690 con 0x7f4578038dc0 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: dump: 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457f59e640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4574001860 con 0x7f4578038dc0 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 1 -- 172.21.15.138:0/2194278158 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4574001c70 con 0x7f4578038dc0 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457d59a640 10 monclient: dump: 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.270+0000 7f457f59e640 4 set_mon_vals no callback set 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.271+0000 7f457f59e640 10 monclient: shutdown 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.271+0000 7f457f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.271+0000 7f457f59e640 1 -- 172.21.15.138:0/2194278158 >> v1:172.21.15.138:6790/0 conn(0x7f4578038dc0 legacy=0x7f45780cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.274+0000 7f457f59e640 1 -- 172.21.15.138:0/2194278158 shutdown_connections 2024-03-29T11:26:43.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.274+0000 7f457f59e640 1 -- 172.21.15.138:0/2194278158 >> 172.21.15.138:0/2194278158 conn(0x7f4578005c00 msgr2=0x7f4578006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:43.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.275+0000 7f457f59e640 1 -- 172.21.15.138:0/2194278158 shutdown_connections 2024-03-29T11:26:43.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.275+0000 7f457f59e640 1 -- 172.21.15.138:0/2194278158 wait complete. 2024-03-29T11:26:43.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.276+0000 7f457f59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:43.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.276+0000 7f457f59e640 10 monclient: monmap: 2024-03-29T11:26:43.263 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:43.263 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:43.263 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:43.263 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.277+0000 7f457f59e640 1 librados: starting msgr at 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.277+0000 7f457f59e640 1 librados: starting objecter 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.277+0000 7f457f59e640 1 Processor -- start 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.277+0000 7f457f59e640 1 -- start start 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.277+0000 7f457f59e640 1 librados: setting wanted keys 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.277+0000 7f457f59e640 1 librados: calling monclient init 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.278+0000 7f457f59e640 10 monclient: init 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.278+0000 7f457f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.278+0000 7f457f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f4578030660 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f45780389e0 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:43.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f45780c9e90 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:43.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457dd9b640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f45780389e0 0x7f457815c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:49840/0 (socket says 172.21.15.138:49840) 2024-03-29T11:26:43.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457dd9b640 1 -- 172.21.15.138:0/731203994 learned_addr learned my addr 172.21.15.138:0/731203994 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45780c7c70 con 0x7f4578030660 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457802db90 con 0x7f45780c9e90 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457815cd90 con 0x7f45780389e0 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.279+0000 7f457f59e640 10 monclient(hunting): authenticate will time out at 1224.589033s 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 10 client.?.objecter ms_handle_connect 0x7f45780389e0 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654654262 0 0) 0x7f457815cd90 con 0x7f45780389e0 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.280+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560004eb0 con 0x7f45780389e0 2024-03-29T11:26:43.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 10 client.?.objecter ms_handle_connect 0x7f45780c9e90 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1507179200 0 0) 0x7f4560004eb0 con 0x7f45780389e0 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457815cd90 con 0x7f45780389e0 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4574001bb0 con 0x7f45780389e0 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 10 monclient(hunting): dump: 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:43.277 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.283+0000 7f456effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157610580 0 0) 0x7f457802db90 con 0x7f45780c9e90 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560004eb0 con 0x7f45780c9e90 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2735086372 0 0) 0x7f457815cd90 con 0x7f45780389e0 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 >> v1:172.21.15.152:6789/0 conn(0x7f45780c9e90 legacy=0x7f4578167210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 >> v1:172.21.15.138:6789/0 conn(0x7f4578030660 legacy=0x7f45780c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 1 monclient: found mon.c 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457815cf20 con 0x7f45780389e0 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient: _finish_auth 0 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f456effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 10 monclient: _renew_subs 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457815d170 con 0x7f45780389e0 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 10 monclient: _renew_subs 2024-03-29T11:26:43.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4578168b10 con 0x7f45780389e0 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.284+0000 7f457f59e640 1 librados: init done 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.285+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4574004b00 con 0x7f45780389e0 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.285+0000 7f456effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.285+0000 7f457cd99640 4 set_mon_vals no callback set 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.285+0000 7f457f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.285+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4574001270 con 0x7f45780389e0 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.286+0000 7f457f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.287+0000 7f457f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.287+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4578007b90 con 0x7f45780389e0 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.287+0000 7f456effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.287+0000 7f456effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.287+0000 7f456effd640 10 monclient: dump: 2024-03-29T11:26:43.280 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.287+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f4574004e30 con 0x7f45780389e0 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.288+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f4574084bf0 con 0x7f45780389e0 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.288+0000 7f456effd640 10 client.4505.objecter ms_dispatch 0x7f4580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.288+0000 7f456effd640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:43.281 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.288+0000 7f456effd640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:43.282 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.288+0000 7f456effd640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:26:43.282 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.288+0000 7f457e59c640 10 monclient: get_auth_request con 0x7f456006dc40 auth_method 0 2024-03-29T11:26:43.282 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.289+0000 7f456effd640 10 client.4505.objecter ms_handle_connect 0x7f456006dc40 2024-03-29T11:26:43.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.297+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4574050300 con 0x7f45780389e0 2024-03-29T11:26:43.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.297+0000 7f456effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:43.284 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.297+0000 7f456effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:43.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.624+0000 7f457f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.624+0000 7f457f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.624+0000 7f457f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:43.613 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.624+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45780c7e20 con 0x7f45780389e0 2024-03-29T11:26:43.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.626+0000 7f456effd640 1 -- 172.21.15.138:0/731203994 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f45780c7e20 con 0x7f45780389e0 2024-03-29T11:26:43.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.627+0000 7f456effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.614 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.627+0000 7f456effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:43.614 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:43.614 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:43.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.630+0000 7f457f59e640 10 librados: watch_flush enter 2024-03-29T11:26:43.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 10 librados: watch_flush exit 2024-03-29T11:26:43.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:26:43.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:26:43.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 >> v1:172.21.15.152:6816/3052585572 conn(0x7f456006dc40 legacy=0x7f4560070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 10 monclient: shutdown 2024-03-29T11:26:43.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:43.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 >> v1:172.21.15.138:6790/0 conn(0x7f45780389e0 legacy=0x7f457815c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 shutdown_connections 2024-03-29T11:26:43.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.631+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 >> 172.21.15.138:0/731203994 conn(0x7f4578005c00 msgr2=0x7f45780c4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:43.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.632+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 shutdown_connections 2024-03-29T11:26:43.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.633+0000 7f457f59e640 1 -- 172.21.15.138:0/731203994 wait complete. 2024-03-29T11:26:43.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:43.633+0000 7f457f59e640 1 librados: shutdown 2024-03-29T11:26:43.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:50.450 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:50.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.612+0000 7f1b001db640 10 monclient: get_monmap_and_config 2024-03-29T11:26:50.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.612+0000 7f1b001db640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.612+0000 7f1b001db640 10 monclient: monmap: 2024-03-29T11:26:50.601 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:50.601 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:50.601 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:50.613910+0000 2024-03-29T11:26:50.601 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:50.613910+0000 2024-03-29T11:26:50.601 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.601 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:50.602 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:50.602 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:50.602 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:50.602 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:50.603 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.613+0000 7f1b001db640 1 Processor -- start 2024-03-29T11:26:50.603 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.613+0000 7f1b001db640 1 -- start start 2024-03-29T11:26:50.603 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.613+0000 7f1b001db640 10 monclient: init 2024-03-29T11:26:50.606 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.614+0000 7f1b001db640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.606 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.614+0000 7f1b001db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:50.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.614+0000 7f1b001db640 10 monclient(hunting): picked mon.noname-a con 0x7f1af80b4100 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:50.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.614+0000 7f1b001db640 10 monclient(hunting): picked mon.noname-c con 0x7f1af813ee90 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:50.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1b001db640 10 monclient(hunting): picked mon.noname-b con 0x7f1af80af900 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:50.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1b001db640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af80afd10 con 0x7f1af80b4100 2024-03-29T11:26:50.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1b001db640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af80af6f0 con 0x7f1af813ee90 2024-03-29T11:26:50.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1b001db640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af80aaf60 con 0x7f1af80af900 2024-03-29T11:26:50.607 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1b001db640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1b001db640 10 monclient(hunting): authenticate will time out at 1231.926873s 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1afe778640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f1af80af900 0x7f1af813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:59270/0 (socket says 172.21.15.138:59270) 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1afe778640 1 -- 172.21.15.138:0/1887252544 learned_addr learned my addr 172.21.15.138:0/1887252544 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.617+0000 7f1afd776640 1 --1- 172.21.15.138:0/1887252544 >> v1:172.21.15.152:6789/0 conn(0x7f1af813ee90 0x7f1af80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:39705/0 (socket says 172.21.15.138:39705) 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.618+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827045304 0 0) 0x7f1af80aaf60 con 0x7f1af80af900 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.618+0000 7f1afcf75640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.618+0000 7f1afcf75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.618+0000 7f1afcf75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.608 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.618+0000 7f1afcf75640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:26:50.609 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.618+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae4003520 con 0x7f1af80af900 2024-03-29T11:26:50.609 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.618+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182324437 0 0) 0x7f1ae4003520 con 0x7f1af80af900 2024-03-29T11:26:50.609 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.619+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 >> v1:172.21.15.152:6789/0 conn(0x7f1af813ee90 legacy=0x7f1af80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.609 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.619+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 >> v1:172.21.15.138:6789/0 conn(0x7f1af80b4100 legacy=0x7f1af80a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 1 monclient: found mon.noname-b 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af813e970 con 0x7f1af80af900 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1af0001a10 con 0x7f1af80af900 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.620+0000 7f1afcf75640 10 monclient: dump: 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:50.620 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.621+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1af0001ce0 con 0x7f1af80af900 2024-03-29T11:26:50.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.621+0000 7f1afcf75640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.621+0000 7f1afcf75640 1 -- 172.21.15.138:0/1887252544 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1af0002070 con 0x7f1af80af900 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.621+0000 7f1afcf75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.621+0000 7f1afcf75640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.621+0000 7f1afcf75640 10 monclient: dump: 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:50.622 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.622+0000 7f1b001db640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.622+0000 7f1b001db640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.622+0000 7f1b001db640 4 set_mon_vals no callback set 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.623+0000 7f1b001db640 10 monclient: shutdown 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.623+0000 7f1b001db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.623+0000 7f1b001db640 1 -- 172.21.15.138:0/1887252544 >> v1:172.21.15.138:6790/0 conn(0x7f1af80af900 legacy=0x7f1af813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.624+0000 7f1b001db640 1 -- 172.21.15.138:0/1887252544 shutdown_connections 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.624+0000 7f1b001db640 1 -- 172.21.15.138:0/1887252544 >> 172.21.15.138:0/1887252544 conn(0x7f1af80b35e0 msgr2=0x7f1af80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.625+0000 7f1b001db640 1 -- 172.21.15.138:0/1887252544 shutdown_connections 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.625+0000 7f1b001db640 1 -- 172.21.15.138:0/1887252544 wait complete. 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 10 monclient: monmap: 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 1 librados: starting msgr at 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 1 librados: starting objecter 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 1 Processor -- start 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 1 -- start start 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 1 librados: setting wanted keys 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 1 librados: calling monclient init 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.626+0000 7f1b001db640 10 monclient: init 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.627+0000 7f1b001db640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.627+0000 7f1b001db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:50.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.627+0000 7f1b001db640 10 monclient(hunting): picked mon.noname-a con 0x7f1af80af900 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.629+0000 7f1b001db640 10 monclient(hunting): picked mon.noname-b con 0x7f1af80b4100 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.629+0000 7f1b001db640 10 monclient(hunting): picked mon.noname-c con 0x7f1af813ee90 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.630+0000 7f1b001db640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af8140120 con 0x7f1af80af900 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.630+0000 7f1b001db640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af8140400 con 0x7f1af80b4100 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.630+0000 7f1b001db640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af8141030 con 0x7f1af813ee90 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.630+0000 7f1b001db640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.630+0000 7f1b001db640 10 monclient(hunting): authenticate will time out at 1231.939093s 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.630+0000 7f1afd776640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f1af80b4100 0x7f1af8135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:39795/0 (socket says 172.21.15.138:39795) 2024-03-29T11:26:50.627 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.630+0000 7f1afd776640 1 -- 172.21.15.138:0/2755820397 learned_addr learned my addr 172.21.15.138:0/2755820397 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 client.?.objecter ms_handle_connect 0x7f1af80af900 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 client.?.objecter ms_handle_connect 0x7f1af813ee90 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394476421 0 0) 0x7f1af8141030 con 0x7f1af813ee90 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1adc004eb0 con 0x7f1af813ee90 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80767862 0 0) 0x7f1af8140120 con 0x7f1af80af900 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af8141030 con 0x7f1af80af900 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 client.?.objecter ms_handle_connect 0x7f1af80b4100 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3925905949 0 0) 0x7f1adc004eb0 con 0x7f1af813ee90 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af8140120 con 0x7f1af813ee90 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1af0001830 con 0x7f1af813ee90 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:50.632 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3601229597 0 0) 0x7f1af8141030 con 0x7f1af80af900 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1adc004eb0 con 0x7f1af80af900 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1af4001830 con 0x7f1af80af900 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:50.633 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1596143205 0 0) 0x7f1adc004eb0 con 0x7f1af80af900 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 >> v1:172.21.15.138:6790/0 conn(0x7f1af813ee90 legacy=0x7f1af81e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 >> v1:172.21.15.152:6789/0 conn(0x7f1af80b4100 legacy=0x7f1af8135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 monclient: found mon.a 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af813fc60 con 0x7f1af80af900 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1aee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1b001db640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1b001db640 10 monclient: _renew_subs 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1b001db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.634+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af813ff10 con 0x7f1af80af900 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.635+0000 7f1b001db640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:50.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.635+0000 7f1b001db640 10 monclient: _renew_subs 2024-03-29T11:26:50.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.635+0000 7f1b001db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:50.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.635+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af81dc5e0 con 0x7f1af80af900 2024-03-29T11:26:50.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.635+0000 7f1b001db640 1 librados: init done 2024-03-29T11:26:50.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.637+0000 7f1b001db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.637+0000 7f1b001db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.637+0000 7f1b001db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.637+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af813f3b0 con 0x7f1af80af900 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.638+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1af4001c40 con 0x7f1af80af900 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.638+0000 7f1aee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.638+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1af40037b0 con 0x7f1af80af900 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.638+0000 7f1aee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.638+0000 7f1aee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.638+0000 7f1aee7fc640 10 monclient: dump: 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.637 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.638+0000 7f1aef7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.639+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f1af4019780 con 0x7f1af80af900 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.640+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f1af4083450 con 0x7f1af80af900 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.640+0000 7f1aee7fc640 10 client.4458.objecter ms_dispatch 0x7f1af8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.640+0000 7f1aee7fc640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:50.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.640+0000 7f1aee7fc640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:50.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.640+0000 7f1aee7fc640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T11:26:50.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.641+0000 7f1afd776640 10 monclient: get_auth_request con 0x7f1adc06e2a0 auth_method 0 2024-03-29T11:26:50.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.644+0000 7f1aee7fc640 10 client.4458.objecter ms_handle_connect 0x7f1adc06e2a0 2024-03-29T11:26:50.639 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.646+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1af404ed90 con 0x7f1af80af900 2024-03-29T11:26:50.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.646+0000 7f1aee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.646+0000 7f1aee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:50.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.962+0000 7f1b001db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.962+0000 7f1b001db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.962+0000 7f1b001db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:26:50.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.962+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1af8075f10 con 0x7f1af80af900 2024-03-29T11:26:50.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.963+0000 7f1aee7fc640 1 -- 172.21.15.138:0/2755820397 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f1af404ed90 con 0x7f1af80af900 2024-03-29T11:26:50.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.963+0000 7f1aee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.963+0000 7f1aee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:50.953 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:50.953 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:50.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.967+0000 7f1b001db640 10 librados: watch_flush enter 2024-03-29T11:26:50.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.967+0000 7f1b001db640 10 librados: watch_flush exit 2024-03-29T11:26:50.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.967+0000 7f1b001db640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T11:26:50.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.967+0000 7f1b001db640 10 client.4458.objecter successfully canceled tick 2024-03-29T11:26:50.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.969+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 >> v1:172.21.15.152:6816/3052585572 conn(0x7f1adc06e2a0 legacy=0x7f1adc070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.969+0000 7f1b001db640 10 monclient: shutdown 2024-03-29T11:26:50.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.969+0000 7f1b001db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.969+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 >> v1:172.21.15.138:6789/0 conn(0x7f1af80af900 legacy=0x7f1af8140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.969+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 shutdown_connections 2024-03-29T11:26:50.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.969+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 >> 172.21.15.138:0/2755820397 conn(0x7f1af80b35e0 msgr2=0x7f1af81428b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.970+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 shutdown_connections 2024-03-29T11:26:50.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.970+0000 7f1b001db640 1 -- 172.21.15.138:0/2755820397 wait complete. 2024-03-29T11:26:50.958 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:50.970+0000 7f1b001db640 1 librados: shutdown 2024-03-29T11:26:50.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:57.786 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:57.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.951+0000 7f8fc60a1640 10 monclient: get_monmap_and_config 2024-03-29T11:26:57.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.951+0000 7f8fc60a1640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.951+0000 7f8fc60a1640 10 monclient: monmap: 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:26:57.952408+0000 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:26:57.952408+0000 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:26:57.939 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:57.940 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.952+0000 7f8fc60a1640 1 Processor -- start 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.953+0000 7f8fc60a1640 1 -- start start 2024-03-29T11:26:57.942 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.953+0000 7f8fc60a1640 10 monclient: init 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 10 monclient(hunting): picked mon.noname-b con 0x7f8fc00a9750 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 10 monclient(hunting): picked mon.noname-a con 0x7f8fc00aacf0 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 10 monclient(hunting): picked mon.noname-c con 0x7f8fc00af220 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc00ac2f0 con 0x7f8fc00aacf0 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc00acaf0 con 0x7f8fc00af220 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc00ac710 con 0x7f8fc00a9750 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fc60a1640 10 monclient(hunting): authenticate will time out at 1239.263340s 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fbf7fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f8fc00a9750 0x7f8fc00ad010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:49986/0 (socket says 172.21.15.138:49986) 2024-03-29T11:26:57.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.954+0000 7f8fbf7fe640 1 -- 172.21.15.138:0/341214310 learned_addr learned my addr 172.21.15.138:0/341214310 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:57.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174032731 0 0) 0x7f8fc00ac2f0 con 0x7f8fc00aacf0 2024-03-29T11:26:57.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa8003440 con 0x7f8fc00aacf0 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099179962 0 0) 0x7f8fc00ac710 con 0x7f8fc00a9750 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.958+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc00ac2f0 con 0x7f8fc00a9750 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813851122 0 0) 0x7f8fa8003440 con 0x7f8fc00aacf0 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 >> v1:172.21.15.138:6790/0 conn(0x7f8fc00a9750 legacy=0x7f8fc00ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 >> v1:172.21.15.152:6789/0 conn(0x7f8fc00af220 legacy=0x7f8fc0149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:57.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 monclient: found mon.noname-a 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc00ac8a0 con 0x7f8fc00aacf0 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8fb40015c0 con 0x7f8fc00aacf0 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: dump: 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:57.951 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fc60a1640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fc60a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fb40017a0 con 0x7f8fc00aacf0 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 1 -- 172.21.15.138:0/341214310 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8fb4001bf0 con 0x7f8fc00aacf0 2024-03-29T11:26:57.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:57.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fbe7fc640 10 monclient: dump: 2024-03-29T11:26:57.955 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:57.955 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:57.955 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fc60a1640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.959+0000 7f8fc60a1640 4 set_mon_vals no callback set 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.960+0000 7f8fc60a1640 10 monclient: shutdown 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.960+0000 7f8fc60a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:57.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.960+0000 7f8fc60a1640 1 -- 172.21.15.138:0/341214310 >> v1:172.21.15.138:6789/0 conn(0x7f8fc00aacf0 legacy=0x7f8fc013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.961+0000 7f8fc60a1640 1 -- 172.21.15.138:0/341214310 shutdown_connections 2024-03-29T11:26:57.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.961+0000 7f8fc60a1640 1 -- 172.21.15.138:0/341214310 >> 172.21.15.138:0/341214310 conn(0x7f8fc006b660 msgr2=0x7f8fc006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:57.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.961+0000 7f8fc60a1640 1 -- 172.21.15.138:0/341214310 shutdown_connections 2024-03-29T11:26:57.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.962+0000 7f8fc60a1640 1 -- 172.21.15.138:0/341214310 wait complete. 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.963+0000 7f8fc60a1640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.963+0000 7f8fc60a1640 10 monclient: monmap: 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.965 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:57.966 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:26:57.966 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:26:57.966 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:26:57.966 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:57.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.963+0000 7f8fc60a1640 1 librados: starting msgr at 2024-03-29T11:26:57.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.963+0000 7f8fc60a1640 1 librados: starting objecter 2024-03-29T11:26:57.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.963+0000 7f8fc60a1640 1 Processor -- start 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.964+0000 7f8fc60a1640 1 -- start start 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.964+0000 7f8fc60a1640 1 librados: setting wanted keys 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.964+0000 7f8fc60a1640 1 librados: calling monclient init 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.964+0000 7f8fc60a1640 10 monclient: init 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.965+0000 7f8fc60a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.965+0000 7f8fc60a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.965+0000 7f8fc60a1640 10 monclient(hunting): picked mon.noname-c con 0x7f8fc00a9750 addr v1:172.21.15.138:6790/0 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.965+0000 7f8fc60a1640 10 monclient(hunting): picked mon.noname-b con 0x7f8fc00aacf0 addr v1:172.21.15.152:6789/0 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.966+0000 7f8fbf7fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f8fc00a9750 0x7f8fc01d65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:50764/0 (socket says 172.21.15.138:50764) 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fc60a1640 10 monclient(hunting): picked mon.noname-a con 0x7f8fc00af220 addr v1:172.21.15.138:6789/0 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fc60a1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc006dae0 con 0x7f8fc00af220 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fc60a1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc0071a00 con 0x7f8fc00aacf0 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fc60a1640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc01d6ce0 con 0x7f8fc00a9750 2024-03-29T11:26:57.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fc60a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fc60a1640 10 monclient(hunting): authenticate will time out at 1239.276156s 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fbf7fe640 1 -- 172.21.15.138:0/1910188063 learned_addr learned my addr 172.21.15.138:0/1910188063 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 10 client.?.objecter ms_handle_connect 0x7f8fc00a9750 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774418029 0 0) 0x7f8fc01d6ce0 con 0x7f8fc00a9750 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.967+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fac004eb0 con 0x7f8fc00a9750 2024-03-29T11:26:57.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053185838 0 0) 0x7f8fac004eb0 con 0x7f8fc00a9750 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fac005300 con 0x7f8fc00a9750 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8fb0001910 con 0x7f8fc00a9750 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 10 monclient(hunting): dump: 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:57.969 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3964834570 0 0) 0x7f8fac005300 con 0x7f8fc00a9750 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.968+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 >> v1:172.21.15.152:6789/0 conn(0x7f8fc00aacf0 legacy=0x7f8fc01348c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 >> v1:172.21.15.138:6789/0 conn(0x7f8fc00af220 legacy=0x7f8fc0134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 1 monclient: found mon.c 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc01d6e70 con 0x7f8fc00a9750 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 10 monclient: _renew_subs 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 1 -- 172.21.15.138:0/1910188063 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc01de8b0 con 0x7f8fc00a9750 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 10 monclient: _renew_subs 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 1 -- 172.21.15.138:0/1910188063 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc01dec20 con 0x7f8fc00a9750 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.969+0000 7f8fc60a1640 1 librados: init done 2024-03-29T11:26:57.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.971+0000 7f8fa5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.971+0000 7f8fa5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.971+0000 7f8fa5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:57.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.971+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa0005080 con 0x7f8fc00a9750 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.978+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fb00047e0 con 0x7f8fc00a9750 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.978+0000 7f8fa7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.978+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8fb0001910 con 0x7f8fc00a9750 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.978+0000 7f8fa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.978+0000 7f8fa7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.978+0000 7f8fa7fff640 10 monclient: dump: 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:26:57.974 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.978+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f8fb001a900 con 0x7f8fc00a9750 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.979+0000 7f8fbd7fa640 4 set_mon_vals no callback set 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.979+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f8fb0084870 con 0x7f8fc00a9750 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.979+0000 7f8fa7fff640 10 client.4517.objecter ms_dispatch 0x7f8fc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:57.975 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.979+0000 7f8fa7fff640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:57.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.979+0000 7f8fa7fff640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:57.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.979+0000 7f8fa7fff640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T11:26:57.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.980+0000 7f8fbeffd640 10 monclient: get_auth_request con 0x7f8fac06e1a0 auth_method 0 2024-03-29T11:26:57.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.981+0000 7f8fa7fff640 10 client.4517.objecter ms_handle_connect 0x7f8fac06e1a0 2024-03-29T11:26:57.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.981+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fb00501d0 con 0x7f8fc00a9750 2024-03-29T11:26:57.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.981+0000 7f8fa7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.976 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:57.981+0000 7f8fa7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:58.297 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.310+0000 7f8fa5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:58.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.310+0000 7f8fa5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:58.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.310+0000 7f8fa5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:26:58.298 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.310+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fa0005330 con 0x7f8fc00a9750 2024-03-29T11:26:58.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.313+0000 7f8fa7fff640 1 -- 172.21.15.138:0/1910188063 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f8fb0052100 con 0x7f8fc00a9750 2024-03-29T11:26:58.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.313+0000 7f8fa7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:58.300 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.313+0000 7f8fa7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:58.300 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:26:58.300 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:58.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 10 librados: watch_flush enter 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 10 librados: watch_flush exit 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 10 client.4517.objecter successfully canceled tick 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 >> v1:172.21.15.152:6816/3052585572 conn(0x7f8fac06e1a0 legacy=0x7f8fac070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 10 monclient: shutdown 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 >> v1:172.21.15.138:6790/0 conn(0x7f8fc00a9750 legacy=0x7f8fc01d65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 shutdown_connections 2024-03-29T11:26:58.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.316+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 >> 172.21.15.138:0/1910188063 conn(0x7f8fc006b660 msgr2=0x7f8fc00b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:58.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.317+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 shutdown_connections 2024-03-29T11:26:58.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.317+0000 7f8fa5ffb640 1 -- 172.21.15.138:0/1910188063 wait complete. 2024-03-29T11:26:58.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:26:58.317+0000 7f8fa5ffb640 1 librados: shutdown 2024-03-29T11:26:58.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:05.131 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:05.289 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.298+0000 7fb02bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.298+0000 7fb02bfff640 10 monclient: build_initial_monmap 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.298+0000 7fb02bfff640 10 monclient: monmap: 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:05.299428+0000 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:05.299428+0000 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:05.290 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.299+0000 7fb02bfff640 1 Processor -- start 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.299+0000 7fb02bfff640 1 -- start start 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.299+0000 7fb02bfff640 10 monclient: init 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fb024029f00 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fb02402b090 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fb0240e2c40 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02402c0f0 con 0x7fb0240e2c40 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02402a430 con 0x7fb02402b090 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02402a5c0 con 0x7fb024029f00 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.300+0000 7fb02bfff640 10 monclient(hunting): authenticate will time out at 1246.609613s 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.301+0000 7fb02affd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fb024029f00 0x7fb0240e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:53928/0 (socket says 172.21.15.138:53928) 2024-03-29T11:27:05.291 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.301+0000 7fb02affd640 1 -- 172.21.15.138:0/1167139291 learned_addr learned my addr 172.21.15.138:0/1167139291 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:05.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.302+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283203761 0 0) 0x7fb02402a5c0 con 0x7fb024029f00 2024-03-29T11:27:05.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.302+0000 7fb029ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:05.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.302+0000 7fb029ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:05.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.302+0000 7fb029ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:05.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.302+0000 7fb029ffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:27:05.292 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.302+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb014003520 con 0x7fb024029f00 2024-03-29T11:27:05.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.304+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073742572 0 0) 0x7fb014003520 con 0x7fb024029f00 2024-03-29T11:27:05.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.304+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 >> v1:172.21.15.152:6789/0 conn(0x7fb02402b090 legacy=0x7fb0240e2530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:05.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 >> v1:172.21.15.138:6789/0 conn(0x7fb0240e2c40 legacy=0x7fb0240f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:05.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:05.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 1 monclient: found mon.noname-b 2024-03-29T11:27:05.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:05.294 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:27:05.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb024006940 con 0x7fb024029f00 2024-03-29T11:27:05.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:05.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:05.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:05.295 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb01c0012f0 con 0x7fb024029f00 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb029ffb640 10 monclient: dump: 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb02bfff640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:27:05.296 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.305+0000 7fb02bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb01c0014c0 con 0x7fb024029f00 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb029ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb029ffb640 1 -- 172.21.15.138:0/1167139291 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb01c0018d0 con 0x7fb024029f00 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:05.301 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb029ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb029ffb640 10 monclient: dump: 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb02bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:05.302 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb02bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.306+0000 7fb02bfff640 10 monclient: shutdown 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.307+0000 7fb02bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.307+0000 7fb02bfff640 1 -- 172.21.15.138:0/1167139291 >> v1:172.21.15.138:6790/0 conn(0x7fb024029f00 legacy=0x7fb0240e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.307+0000 7fb02bfff640 1 -- 172.21.15.138:0/1167139291 shutdown_connections 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.307+0000 7fb02bfff640 1 -- 172.21.15.138:0/1167139291 >> 172.21.15.138:0/1167139291 conn(0x7fb0240058f0 msgr2=0x7fb024005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.307+0000 7fb02bfff640 1 -- 172.21.15.138:0/1167139291 shutdown_connections 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.307+0000 7fb02bfff640 1 -- 172.21.15.138:0/1167139291 wait complete. 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.308+0000 7fb02bfff640 10 monclient: build_initial_monmap 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.308+0000 7fb02bfff640 10 monclient: monmap: 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:05.303 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.308+0000 7fb02bfff640 1 librados: starting msgr at 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.308+0000 7fb02bfff640 1 librados: starting objecter 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.309+0000 7fb02bfff640 1 Processor -- start 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.309+0000 7fb02bfff640 1 -- start start 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.309+0000 7fb02bfff640 1 librados: setting wanted keys 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.309+0000 7fb02bfff640 1 librados: calling monclient init 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.309+0000 7fb02bfff640 10 monclient: init 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:05.304 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fb024029f00 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fb02402b090 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fb0240e2c40 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb024034e60 con 0x7fb024029f00 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb024035380 con 0x7fb02402b090 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0240e6040 con 0x7fb0240e2c40 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.310+0000 7fb02bfff640 10 monclient(hunting): authenticate will time out at 1246.619305s 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.313+0000 7fb02affd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fb024029f00 0x7fb024034670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:41637/0 (socket says 172.21.15.138:41637) 2024-03-29T11:27:05.305 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.313+0000 7fb02affd640 1 -- 172.21.15.138:0/2672567108 learned_addr learned my addr 172.21.15.138:0/2672567108 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:05.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 client.?.objecter ms_handle_connect 0x7fb024029f00 2024-03-29T11:27:05.308 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 client.?.objecter ms_handle_connect 0x7fb0240e2c40 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231472203 0 0) 0x7fb0240e6040 con 0x7fb0240e2c40 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb018004eb0 con 0x7fb0240e2c40 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436047408 0 0) 0x7fb024034e60 con 0x7fb024029f00 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:27:05.309 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.314+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0240e6040 con 0x7fb024029f00 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2899650001 0 0) 0x7fb0240e6040 con 0x7fb024029f00 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb024034e60 con 0x7fb024029f00 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb01c001d30 con 0x7fb024029f00 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 10 monclient(hunting): dump: 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:05.310 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1087029351 0 0) 0x7fb024034e60 con 0x7fb024029f00 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 >> v1:172.21.15.138:6790/0 conn(0x7fb0240e2c40 legacy=0x7fb0240cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 >> v1:172.21.15.152:6789/0 conn(0x7fb02402b090 legacy=0x7fb024182000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 monclient: found mon.a 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.315+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0240e61d0 con 0x7fb024029f00 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb013fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:05.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb013fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb013fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 10 monclient: _renew_subs 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb024182710 con 0x7fb024029f00 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 10 monclient: _renew_subs 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:05.312 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0240d05b0 con 0x7fb024029f00 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.316+0000 7fb02bfff640 1 librados: init done 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.318+0000 7fb02bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.318+0000 7fb02bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.318+0000 7fb02bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.318+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb024034ff0 con 0x7fb024029f00 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.319+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb01c000ed0 con 0x7fb024029f00 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.319+0000 7fb013fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.319+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fb01c004dd0 con 0x7fb024029f00 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.319+0000 7fb013fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.319+0000 7fb013fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.319+0000 7fb013fff640 10 monclient: dump: 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.313 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.320+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fb01c01ada0 con 0x7fb024029f00 2024-03-29T11:27:05.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.321+0000 7fb028ff9640 4 set_mon_vals no callback set 2024-03-29T11:27:05.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.322+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fb01c084ca0 con 0x7fb024029f00 2024-03-29T11:27:05.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.322+0000 7fb013fff640 10 client.4464.objecter ms_dispatch 0x7fb02c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:05.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.322+0000 7fb013fff640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:05.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.322+0000 7fb013fff640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:05.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.322+0000 7fb013fff640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T11:27:05.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.323+0000 7fb02a7fc640 10 monclient: get_auth_request con 0x7fb01806dd30 auth_method 0 2024-03-29T11:27:05.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.324+0000 7fb013fff640 10 client.4464.objecter ms_handle_connect 0x7fb01806dd30 2024-03-29T11:27:05.319 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.332+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb01c0506c0 con 0x7fb024029f00 2024-03-29T11:27:05.320 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.332+0000 7fb013fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:05.320 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.332+0000 7fb013fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:05.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.650+0000 7fb02bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.650+0000 7fb02bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.650+0000 7fb02bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:05.638 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.650+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb024004cd0 con 0x7fb024029f00 2024-03-29T11:27:05.640 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:05.640 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:05.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.651+0000 7fb013fff640 1 -- 172.21.15.138:0/2672567108 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fb01c0525f0 con 0x7fb024029f00 2024-03-29T11:27:05.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.651+0000 7fb013fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.640 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.651+0000 7fb013fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:05.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.655+0000 7fb02bfff640 10 librados: watch_flush enter 2024-03-29T11:27:05.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.655+0000 7fb02bfff640 10 librados: watch_flush exit 2024-03-29T11:27:05.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.655+0000 7fb02bfff640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T11:27:05.643 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.656+0000 7fb02bfff640 10 client.4464.objecter successfully canceled tick 2024-03-29T11:27:05.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.657+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 >> v1:172.21.15.152:6816/3052585572 conn(0x7fb01806dd30 legacy=0x7fb0180701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.657+0000 7fb02bfff640 10 monclient: shutdown 2024-03-29T11:27:05.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.657+0000 7fb02bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:05.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.657+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 >> v1:172.21.15.138:6789/0 conn(0x7fb024029f00 legacy=0x7fb024034670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.657+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 shutdown_connections 2024-03-29T11:27:05.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.657+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 >> 172.21.15.138:0/2672567108 conn(0x7fb0240058f0 msgr2=0x7fb0240e2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:05.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.658+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 shutdown_connections 2024-03-29T11:27:05.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.659+0000 7fb02bfff640 1 -- 172.21.15.138:0/2672567108 wait complete. 2024-03-29T11:27:05.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:05.659+0000 7fb02bfff640 1 librados: shutdown 2024-03-29T11:27:05.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:12.472 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:12.625 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.637+0000 7fa3a6984640 10 monclient: get_monmap_and_config 2024-03-29T11:27:12.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.637+0000 7fa3a6984640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.626 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.637+0000 7fa3a6984640 10 monclient: monmap: 2024-03-29T11:27:12.626 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:12.626 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:12.626 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:12.638903+0000 2024-03-29T11:27:12.627 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:12.638903+0000 2024-03-29T11:27:12.627 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.627 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:12.627 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:12.627 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:12.627 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:12.627 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:12.628 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.640+0000 7fa3a6984640 1 Processor -- start 2024-03-29T11:27:12.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.643+0000 7fa3a6984640 1 -- start start 2024-03-29T11:27:12.631 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.643+0000 7fa3a6984640 10 monclient: init 2024-03-29T11:27:12.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:12.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a006ade0 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:12.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 10 monclient(hunting): picked mon.noname-c con 0x7fa3a00afa90 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:12.634 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 10 monclient(hunting): picked mon.noname-b con 0x7fa3a00ae550 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:12.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a00b0b20 con 0x7fa3a006ade0 2024-03-29T11:27:12.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a013d8c0 con 0x7fa3a00afa90 2024-03-29T11:27:12.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a013da50 con 0x7fa3a00ae550 2024-03-29T11:27:12.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.635 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.644+0000 7fa3a6984640 10 monclient(hunting): authenticate will time out at 1253.953516s 2024-03-29T11:27:12.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.647+0000 7fa3a4f21640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fa3a00ae550 0x7fa3a014b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:55948/0 (socket says 172.21.15.138:55948) 2024-03-29T11:27:12.636 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.647+0000 7fa3a4f21640 1 -- 172.21.15.138:0/1661995293 learned_addr learned my addr 172.21.15.138:0/1661995293 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:12.644 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559050336 0 0) 0x7fa3a013d8c0 con 0x7fa3a00afa90 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa388003520 con 0x7fa3a00afa90 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587761706 0 0) 0x7fa3a00b0b20 con 0x7fa3a006ade0 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.645 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a013d8c0 con 0x7fa3a006ade0 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2198508640 0 0) 0x7fa388003520 con 0x7fa3a00afa90 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 >> v1:172.21.15.138:6790/0 conn(0x7fa3a00ae550 legacy=0x7fa3a014b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 >> v1:172.21.15.138:6789/0 conn(0x7fa3a006ade0 legacy=0x7fa3a0143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 monclient: found mon.noname-c 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0072570 con 0x7fa3a00afa90 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.646 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa398001a10 con 0x7fa3a00afa90 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa39effd640 10 monclient: dump: 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:12.647 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa3a6984640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.649+0000 7fa3a6984640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa398001b70 con 0x7fa3a00afa90 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa39effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa39effd640 1 -- 172.21.15.138:0/1661995293 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa398001fc0 con 0x7fa3a00afa90 2024-03-29T11:27:12.648 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa3a6984640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:12.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa3a6984640 4 set_mon_vals no callback set 2024-03-29T11:27:12.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa39effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:12.649 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.650+0000 7fa39effd640 10 monclient: dump: 2024-03-29T11:27:12.649 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:12.649 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:12.650 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.650 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.650 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.650 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:12.650 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:12.650 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:12.650 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:12.651 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:12.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.651+0000 7fa3a6984640 10 monclient: shutdown 2024-03-29T11:27:12.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.651+0000 7fa3a6984640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.651+0000 7fa3a6984640 1 -- 172.21.15.138:0/1661995293 >> v1:172.21.15.152:6789/0 conn(0x7fa3a00afa90 legacy=0x7fa3a013d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.651 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.651+0000 7fa3a6984640 1 -- 172.21.15.138:0/1661995293 shutdown_connections 2024-03-29T11:27:12.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.651+0000 7fa3a6984640 1 -- 172.21.15.138:0/1661995293 >> 172.21.15.138:0/1661995293 conn(0x7fa3a00b4930 msgr2=0x7fa3a00b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.651+0000 7fa3a6984640 1 -- 172.21.15.138:0/1661995293 shutdown_connections 2024-03-29T11:27:12.652 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.651+0000 7fa3a6984640 1 -- 172.21.15.138:0/1661995293 wait complete. 2024-03-29T11:27:12.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.652+0000 7fa3a6984640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.653 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.652+0000 7fa3a6984640 10 monclient: monmap: 2024-03-29T11:27:12.653 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:12.654 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:12.654 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:12.654 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:12.654 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.654 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:12.654 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:12.655 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:12.655 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:12.655 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:12.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.653+0000 7fa3a6984640 1 librados: starting msgr at 2024-03-29T11:27:12.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.653+0000 7fa3a6984640 1 librados: starting objecter 2024-03-29T11:27:12.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.653+0000 7fa3a6984640 1 Processor -- start 2024-03-29T11:27:12.655 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.653+0000 7fa3a6984640 1 -- start start 2024-03-29T11:27:12.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.653+0000 7fa3a6984640 1 librados: setting wanted keys 2024-03-29T11:27:12.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.653+0000 7fa3a6984640 1 librados: calling monclient init 2024-03-29T11:27:12.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.653+0000 7fa3a6984640 10 monclient: init 2024-03-29T11:27:12.656 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:12.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a006ade0 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:12.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 10 monclient(hunting): picked mon.noname-b con 0x7fa3a00ae550 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:12.657 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 10 monclient(hunting): picked mon.noname-c con 0x7fa3a00afa90 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a013a3b0 con 0x7fa3a006ade0 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a0139120 con 0x7fa3a00ae550 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a01392b0 con 0x7fa3a00afa90 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.654+0000 7fa3a6984640 10 monclient(hunting): authenticate will time out at 1253.963442s 2024-03-29T11:27:12.658 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.655+0000 7fa3a4f21640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fa3a00afa90 0x7fa3a01db4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:58782/0 (socket says 172.21.15.138:58782) 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.655+0000 7fa3a4f21640 1 -- 172.21.15.138:0/3360074036 learned_addr learned my addr 172.21.15.138:0/3360074036 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.655+0000 7fa39cff9640 10 client.?.objecter ms_handle_connect 0x7fa3a00afa90 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.655+0000 7fa39cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.656+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413786510 0 0) 0x7fa3a01392b0 con 0x7fa3a00afa90 2024-03-29T11:27:12.659 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.656+0000 7fa39cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.656+0000 7fa39cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.656+0000 7fa39cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.656+0000 7fa39cff9640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:27:12.660 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.656+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa38c004eb0 con 0x7fa3a00afa90 2024-03-29T11:27:12.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4004495381 0 0) 0x7fa38c004eb0 con 0x7fa3a00afa90 2024-03-29T11:27:12.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3a01392b0 con 0x7fa3a00afa90 2024-03-29T11:27:12.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa390001830 con 0x7fa3a00afa90 2024-03-29T11:27:12.665 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:12.666 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient(hunting): dump: 2024-03-29T11:27:12.666 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:12.666 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:12.666 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.667 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.667 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.667 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:12.667 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:12.667 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:12.667 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:12.668 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:12.668 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:12.668 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 client.?.objecter ms_handle_connect 0x7fa3a006ade0 2024-03-29T11:27:12.668 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.668 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1746374316 0 0) 0x7fa3a01392b0 con 0x7fa3a00afa90 2024-03-29T11:27:12.668 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 >> v1:172.21.15.152:6789/0 conn(0x7fa3a00ae550 legacy=0x7fa3a01345b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:27:12.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 >> v1:172.21.15.138:6789/0 conn(0x7fa3a006ade0 legacy=0x7fa3a0139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 monclient: found mon.c 2024-03-29T11:27:12.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:12.669 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0134cc0 con 0x7fa3a00afa90 2024-03-29T11:27:12.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3900047e0 con 0x7fa3a00afa90 2024-03-29T11:27:12.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.670 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fa390001830 con 0x7fa3a00afa90 2024-03-29T11:27:12.671 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.671 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:12.671 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39cff9640 10 monclient: dump: 2024-03-29T11:27:12.671 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:12.671 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:12.671 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.671 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:12.672 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.672 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:12.672 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:12.672 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:12.672 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:12.673 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:12.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.658+0000 7fa39e7fc640 4 set_mon_vals no callback set 2024-03-29T11:27:12.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:27:12.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 10 monclient: _renew_subs 2024-03-29T11:27:12.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:12.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0134f10 con 0x7fa3a00afa90 2024-03-29T11:27:12.673 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:12.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 10 monclient: _renew_subs 2024-03-29T11:27:12.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:12.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a0070eb0 con 0x7fa3a00afa90 2024-03-29T11:27:12.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.659+0000 7fa3a6984640 1 librados: init done 2024-03-29T11:27:12.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fa390004a90 con 0x7fa3a00afa90 2024-03-29T11:27:12.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa3a6984640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.674 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa3a6984640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa3a6984640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a0075f10 con 0x7fa3a00afa90 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fa3900846f0 con 0x7fa3a00afa90 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa39cff9640 10 client.4529.objecter ms_dispatch 0x7fa3a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa39cff9640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa39cff9640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.662+0000 7fa39cff9640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.663+0000 7fa39ffff640 10 monclient: get_auth_request con 0x7fa38c06e090 auth_method 0 2024-03-29T11:27:12.675 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.664+0000 7fa39cff9640 10 client.4529.objecter ms_handle_connect 0x7fa38c06e090 2024-03-29T11:27:12.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.672+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa39004ffd0 con 0x7fa3a00afa90 2024-03-29T11:27:12.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.672+0000 7fa39cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.676 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.672+0000 7fa39cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:12.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.994+0000 7fa3a6984640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.994+0000 7fa3a6984640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.994+0000 7fa3a6984640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:12.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.994+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3a013a160 con 0x7fa3a00afa90 2024-03-29T11:27:12.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.996+0000 7fa39cff9640 1 -- 172.21.15.138:0/3360074036 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fa390001690 con 0x7fa3a00afa90 2024-03-29T11:27:12.984 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:12.985 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.996+0000 7fa39cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.985 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:12.996+0000 7fa39cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:12.985 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:12.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 10 librados: watch_flush enter 2024-03-29T11:27:12.988 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 10 librados: watch_flush exit 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 10 client.4529.objecter successfully canceled tick 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 >> v1:172.21.15.152:6816/3052585572 conn(0x7fa38c06e090 legacy=0x7fa38c070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 10 monclient: shutdown 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 >> v1:172.21.15.138:6790/0 conn(0x7fa3a00afa90 legacy=0x7fa3a01db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.000+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 shutdown_connections 2024-03-29T11:27:12.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.001+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 >> 172.21.15.138:0/3360074036 conn(0x7fa3a00b4930 msgr2=0x7fa3a00b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.004+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 shutdown_connections 2024-03-29T11:27:12.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.004+0000 7fa3a6984640 1 -- 172.21.15.138:0/3360074036 wait complete. 2024-03-29T11:27:12.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:13.004+0000 7fa3a6984640 1 librados: shutdown 2024-03-29T11:27:13.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:19.818 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:19.979 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.991+0000 7f98ee558640 10 monclient: get_monmap_and_config 2024-03-29T11:27:19.980 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.991+0000 7f98ee558640 10 monclient: build_initial_monmap 2024-03-29T11:27:19.980 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.991+0000 7f98ee558640 10 monclient: monmap: 2024-03-29T11:27:19.980 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:19.980 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:19.980 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:19.991860+0000 2024-03-29T11:27:19.980 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:19.991860+0000 2024-03-29T11:27:19.980 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:19.981 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:19.981 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:19.981 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:19.981 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:19.981 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.993+0000 7f98ee558640 1 Processor -- start 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.993+0000 7f98ee558640 1 -- start start 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.993+0000 7f98ee558640 10 monclient: init 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 10 monclient(hunting): picked mon.noname-c con 0x7f98e80ae250 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 10 monclient(hunting): picked mon.noname-a con 0x7f98e813f970 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:19.982 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 10 monclient(hunting): picked mon.noname-b con 0x7f98e8142800 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:19.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e80afe00 con 0x7f98e813f970 2024-03-29T11:27:19.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e8142c10 con 0x7f98e80ae250 2024-03-29T11:27:19.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e80afb20 con 0x7f98e8142800 2024-03-29T11:27:19.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:19.983 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.994+0000 7f98ee558640 10 monclient(hunting): authenticate will time out at 1261.302575s 2024-03-29T11:27:19.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.997+0000 7f98e7fff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f98e80ae250 0x7f98e813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:54332/0 (socket says 172.21.15.138:54332) 2024-03-29T11:27:19.984 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.997+0000 7f98e7fff640 1 -- 172.21.15.138:0/109734034 learned_addr learned my addr 172.21.15.138:0/109734034 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:19.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719012333 0 0) 0x7f98e8142c10 con 0x7f98e80ae250 2024-03-29T11:27:19.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.989 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:27:19.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d0003520 con 0x7f98e80ae250 2024-03-29T11:27:19.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943225204 0 0) 0x7f98d0003520 con 0x7f98e80ae250 2024-03-29T11:27:19.990 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 >> v1:172.21.15.138:6790/0 conn(0x7f98e8142800 legacy=0x7f98e814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:19.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 >> v1:172.21.15.138:6789/0 conn(0x7f98e813f970 legacy=0x7f98e81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:19.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:19.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:27:19.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:19.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:27:19.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.998+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80aab80 con 0x7f98e80ae250 2024-03-29T11:27:19.991 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:19.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:19.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:19.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f98e0001a10 con 0x7f98e80ae250 2024-03-29T11:27:19.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:19.992 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 10 monclient: dump: 2024-03-29T11:27:19.993 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:19.993 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:19.993 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:19.993 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:19.993 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.993 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:19.994 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:19.994 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:19.994 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:19.994 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:19.994 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98ee558640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:27:19.994 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98ee558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:19.994 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98e0001ca0 con 0x7f98e80ae250 2024-03-29T11:27:19.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:19.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98ee558640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:19.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:19.999+0000 7f98ee558640 4 set_mon_vals no callback set 2024-03-29T11:27:19.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98e6ffd640 1 -- 172.21.15.138:0/109734034 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f98e0002030 con 0x7f98e80ae250 2024-03-29T11:27:19.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.995 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98e6ffd640 10 monclient: dump: 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:19.996 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98ee558640 10 monclient: shutdown 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98ee558640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98ee558640 1 -- 172.21.15.138:0/109734034 >> v1:172.21.15.152:6789/0 conn(0x7f98e80ae250 legacy=0x7f98e813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98ee558640 1 -- 172.21.15.138:0/109734034 shutdown_connections 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.000+0000 7f98ee558640 1 -- 172.21.15.138:0/109734034 >> 172.21.15.138:0/109734034 conn(0x7f98e80b35e0 msgr2=0x7f98e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.001+0000 7f98ee558640 1 -- 172.21.15.138:0/109734034 shutdown_connections 2024-03-29T11:27:19.997 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.001+0000 7f98ee558640 1 -- 172.21.15.138:0/109734034 wait complete. 2024-03-29T11:27:20.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.002+0000 7f98ee558640 10 monclient: build_initial_monmap 2024-03-29T11:27:20.003 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.002+0000 7f98ee558640 10 monclient: monmap: 2024-03-29T11:27:20.003 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:20.003 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.002+0000 7f98ee558640 1 librados: starting msgr at 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.002+0000 7f98ee558640 1 librados: starting objecter 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.002+0000 7f98ee558640 1 Processor -- start 2024-03-29T11:27:20.004 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.003+0000 7f98ee558640 1 -- start start 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.003+0000 7f98ee558640 1 librados: setting wanted keys 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.003+0000 7f98ee558640 1 librados: calling monclient init 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.003+0000 7f98ee558640 10 monclient: init 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.003+0000 7f98ee558640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.003+0000 7f98ee558640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.003+0000 7f98ee558640 10 monclient(hunting): picked mon.noname-b con 0x7f98e813f970 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.004+0000 7f98ee558640 10 monclient(hunting): picked mon.noname-a con 0x7f98e8142800 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.004+0000 7f98ee558640 10 monclient(hunting): picked mon.noname-c con 0x7f98e8140c90 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.004+0000 7f98ee558640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e81464d0 con 0x7f98e8142800 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.004+0000 7f98ee558640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e8141200 con 0x7f98e813f970 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.004+0000 7f98ee558640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e81447a0 con 0x7f98e8140c90 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.004+0000 7f98ee558640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.004+0000 7f98ee558640 10 monclient(hunting): authenticate will time out at 1261.312117s 2024-03-29T11:27:20.005 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.008+0000 7f98ecaf5640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f98e8140c90 0x7f98e81e5fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:51559/0 (socket says 172.21.15.138:51559) 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.008+0000 7f98ecaf5640 1 -- 172.21.15.138:0/1203186799 learned_addr learned my addr 172.21.15.138:0/1203186799 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.009+0000 7f98e4ff9640 10 client.?.objecter ms_handle_connect 0x7f98e8142800 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.009+0000 7f98e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.009+0000 7f98e4ff9640 10 client.?.objecter ms_handle_connect 0x7f98e8140c90 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.009+0000 7f98e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 client.?.objecter ms_handle_connect 0x7f98e813f970 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323590228 0 0) 0x7f98e81464d0 con 0x7f98e8142800 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d4004eb0 con 0x7f98e8142800 2024-03-29T11:27:20.006 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610202675 0 0) 0x7f98e81447a0 con 0x7f98e8140c90 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e81464d0 con 0x7f98e8140c90 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230086249 0 0) 0x7f98e8141200 con 0x7f98e813f970 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e81447a0 con 0x7f98e813f970 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090318224 0 0) 0x7f98d4004eb0 con 0x7f98e8142800 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98e8141200 con 0x7f98e8142800 2024-03-29T11:27:20.007 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f98d8001510 con 0x7f98e8142800 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:20.008 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.010+0000 7f98e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2459933373 0 0) 0x7f98e81464d0 con 0x7f98e8140c90 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d4004eb0 con 0x7f98e8140c90 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f98dc001600 con 0x7f98e8140c90 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:20.009 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1056214950 0 0) 0x7f98e8141200 con 0x7f98e8142800 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 >> v1:172.21.15.138:6790/0 conn(0x7f98e8140c90 legacy=0x7f98e81e5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 >> v1:172.21.15.152:6789/0 conn(0x7f98e813f970 legacy=0x7f98e8143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 monclient: found mon.a 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8144930 con 0x7f98e8142800 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:20.010 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 10 monclient: _renew_subs 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e8145290 con 0x7f98e8142800 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 10 monclient: _renew_subs 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e81e76c0 con 0x7f98e8142800 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.011+0000 7f98ee558640 1 librados: init done 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.014+0000 7f98ee558640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.014+0000 7f98ee558640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.014+0000 7f98ee558640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.014+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e8075f10 con 0x7f98e8142800 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.015+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d8001a20 con 0x7f98e8142800 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.015+0000 7f98e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:20.011 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.015+0000 7f98e67fc640 4 set_mon_vals no callback set 2024-03-29T11:27:20.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.017+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f98d8003910 con 0x7f98e8142800 2024-03-29T11:27:20.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.017+0000 7f98e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.017+0000 7f98e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:20.012 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.017+0000 7f98e4ff9640 10 monclient: dump: 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.017+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f98d80198e0 con 0x7f98e8142800 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.018+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f98d8083760 con 0x7f98e8142800 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.018+0000 7f98e4ff9640 10 client.4470.objecter ms_dispatch 0x7f98e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:20.013 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.018+0000 7f98e4ff9640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:20.014 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.018+0000 7f98e4ff9640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:20.014 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.018+0000 7f98e4ff9640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T11:27:20.014 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.018+0000 7f98e7fff640 10 monclient: get_auth_request con 0x7f98d406e030 auth_method 0 2024-03-29T11:27:20.014 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.019+0000 7f98e4ff9640 10 client.4470.objecter ms_handle_connect 0x7f98d406e030 2024-03-29T11:27:20.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.028+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98d804f1b0 con 0x7f98e8142800 2024-03-29T11:27:20.015 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.028+0000 7f98e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.016 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.028+0000 7f98e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:20.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.390+0000 7f98ee558640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.390+0000 7f98ee558640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.390+0000 7f98ee558640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:20.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.390+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98e8144ac0 con 0x7f98e8142800 2024-03-29T11:27:20.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.393+0000 7f98e4ff9640 1 -- 172.21.15.138:0/1203186799 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f98d80510e0 con 0x7f98e8142800 2024-03-29T11:27:20.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.393+0000 7f98e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.393+0000 7f98e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:20.384 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:20.384 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:20.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 10 librados: watch_flush enter 2024-03-29T11:27:20.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 10 librados: watch_flush exit 2024-03-29T11:27:20.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T11:27:20.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 10 client.4470.objecter successfully canceled tick 2024-03-29T11:27:20.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 >> v1:172.21.15.152:6816/3052585572 conn(0x7f98d406e030 legacy=0x7f98d40704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 10 monclient: shutdown 2024-03-29T11:27:20.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:20.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.397+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 >> v1:172.21.15.138:6789/0 conn(0x7f98e8142800 legacy=0x7f98e8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.398+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 shutdown_connections 2024-03-29T11:27:20.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.399+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 >> 172.21.15.138:0/1203186799 conn(0x7f98e80b35e0 msgr2=0x7f98e80a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:20.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.399+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 shutdown_connections 2024-03-29T11:27:20.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.400+0000 7f98ee558640 1 -- 172.21.15.138:0/1203186799 wait complete. 2024-03-29T11:27:20.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:20.400+0000 7f98ee558640 1 librados: shutdown 2024-03-29T11:27:20.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:27.220 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:27.372 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.385+0000 7f1cf6f7b640 10 monclient: get_monmap_and_config 2024-03-29T11:27:27.372 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.385+0000 7f1cf6f7b640 10 monclient: build_initial_monmap 2024-03-29T11:27:27.372 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.385+0000 7f1cf6f7b640 10 monclient: monmap: 2024-03-29T11:27:27.372 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:27.385350+0000 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:27.385350+0000 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:27.373 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.386+0000 7f1cf6f7b640 1 Processor -- start 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.386+0000 7f1cf6f7b640 1 -- start start 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.386+0000 7f1cf6f7b640 10 monclient: init 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf01457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf013f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:27.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf00a9d70 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:27.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf00aaef0 con 0x7f1cf01457f0 2024-03-29T11:27:27.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf0145bd0 con 0x7f1cf00a9d70 2024-03-29T11:27:27.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf0145eb0 con 0x7f1cf013f650 2024-03-29T11:27:27.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:27.375 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf6f7b640 10 monclient(hunting): authenticate will time out at 1268.695168s 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf4d17640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f1cf01457f0 0x7f1cf013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:35856/0 (socket says 172.21.15.138:35856) 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1cf4d17640 1 -- 172.21.15.138:0/123199389 learned_addr learned my addr 172.21.15.138:0/123199389 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.387+0000 7f1ceffff640 1 --1- 172.21.15.138:0/123199389 >> v1:172.21.15.138:6790/0 conn(0x7f1cf013f650 0x7f1cf00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:35788/0 (socket says 172.21.15.138:35788) 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105669293 0 0) 0x7f1cf0145bd0 con 0x7f1cf00a9d70 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce0003520 con 0x7f1cf00a9d70 2024-03-29T11:27:27.378 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060603634 0 0) 0x7f1cf00aaef0 con 0x7f1cf01457f0 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.388+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf0145bd0 con 0x7f1cf01457f0 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922031559 0 0) 0x7f1ce0003520 con 0x7f1cf00a9d70 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 >> v1:172.21.15.138:6790/0 conn(0x7f1cf013f650 legacy=0x7f1cf00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 >> v1:172.21.15.138:6789/0 conn(0x7f1cf01457f0 legacy=0x7f1cf013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 1 monclient: found mon.noname-c 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:27:27.379 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf0146340 con 0x7f1cf00a9d70 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cf6f7b640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cf6f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1ce8001a10 con 0x7f1cf00a9d70 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cef7fe640 10 monclient: dump: 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.380 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.381 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.381 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.382 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:27.382 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:27.382 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:27.382 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.382 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.389+0000 7f1cf6f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:27.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce8001c30 con 0x7f1cf00a9d70 2024-03-29T11:27:27.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:27.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cf6f7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:27.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cf6f7b640 4 set_mon_vals no callback set 2024-03-29T11:27:27.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cef7fe640 1 -- 172.21.15.138:0/123199389 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1ce8001fc0 con 0x7f1cf00a9d70 2024-03-29T11:27:27.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.391+0000 7f1cef7fe640 10 monclient: dump: 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:27.388 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.392+0000 7f1cf6f7b640 10 monclient: shutdown 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.392+0000 7f1cf6f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.392+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/123199389 >> v1:172.21.15.152:6789/0 conn(0x7f1cf00a9d70 legacy=0x7f1cf014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.392+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/123199389 shutdown_connections 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.392+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/123199389 >> 172.21.15.138:0/123199389 conn(0x7f1cf0066a00 msgr2=0x7f1cf0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.392+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/123199389 shutdown_connections 2024-03-29T11:27:27.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.393+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/123199389 wait complete. 2024-03-29T11:27:27.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 10 monclient: build_initial_monmap 2024-03-29T11:27:27.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 10 monclient: monmap: 2024-03-29T11:27:27.390 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:27.390 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 1 librados: starting msgr at 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 1 librados: starting objecter 2024-03-29T11:27:27.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 1 Processor -- start 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 1 -- start start 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 1 librados: setting wanted keys 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 1 librados: calling monclient init 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.394+0000 7f1cf6f7b640 10 monclient: init 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf00a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf013f650 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf01457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf013aec0 con 0x7f1cf01457f0 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf013ba10 con 0x7f1cf013f650 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf0139d30 con 0x7f1cf00a9d70 2024-03-29T11:27:27.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1cf6f7b640 10 monclient(hunting): authenticate will time out at 1268.704812s 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1ceffff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f1cf013f650 0x7f1cf01355c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:48708/0 (socket says 172.21.15.138:48708) 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.396+0000 7f1ceffff640 1 -- 172.21.15.138:0/1043557446 learned_addr learned my addr 172.21.15.138:0/1043557446 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 client.?.objecter ms_handle_connect 0x7f1cf00a9d70 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 client.?.objecter ms_handle_connect 0x7f1cf013f650 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 client.?.objecter ms_handle_connect 0x7f1cf01457f0 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483068566 0 0) 0x7f1cf013ba10 con 0x7f1cf013f650 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4004eb0 con 0x7f1cf013f650 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253481749 0 0) 0x7f1cf0139d30 con 0x7f1cf00a9d70 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf013ba10 con 0x7f1cf00a9d70 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930302580 0 0) 0x7f1cf013aec0 con 0x7f1cf01457f0 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.397+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf0139d30 con 0x7f1cf01457f0 2024-03-29T11:27:27.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2067017843 0 0) 0x7f1cf0139d30 con 0x7f1cf01457f0 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf013aec0 con 0x7f1cf01457f0 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1ce80017b0 con 0x7f1cf01457f0 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 10 monclient(hunting): dump: 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.395 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:27.396 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:27.396 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:27.396 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:27.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1625005653 0 0) 0x7f1cf013ba10 con 0x7f1cf00a9d70 2024-03-29T11:27:27.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf0139d30 con 0x7f1cf00a9d70 2024-03-29T11:27:27.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1ce4001810 con 0x7f1cf00a9d70 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.398+0000 7f1ced7fa640 10 monclient(hunting): dump: 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:27.397 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:27.398 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:27.398 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2420989450 0 0) 0x7f1cd4004eb0 con 0x7f1cf013f650 2024-03-29T11:27:27.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf013ba10 con 0x7f1cf013f650 2024-03-29T11:27:27.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1cdc0019d0 con 0x7f1cf013f650 2024-03-29T11:27:27.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 10 monclient(hunting): dump: 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3377613933 0 0) 0x7f1cf013aec0 con 0x7f1cf01457f0 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 >> v1:172.21.15.138:6790/0 conn(0x7f1cf00a9d70 legacy=0x7f1cf0139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 >> v1:172.21.15.152:6789/0 conn(0x7f1cf013f650 legacy=0x7f1cf01355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 monclient: found mon.a 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf0138b40 con 0x7f1cf01457f0 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1ced7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 10 monclient: _renew_subs 2024-03-29T11:27:27.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:27.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf013b6a0 con 0x7f1cf01457f0 2024-03-29T11:27:27.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:27.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 10 monclient: _renew_subs 2024-03-29T11:27:27.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:27.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf0135eb0 con 0x7f1cf01457f0 2024-03-29T11:27:27.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.399+0000 7f1cf6f7b640 1 librados: init done 2024-03-29T11:27:27.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.400+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce80019f0 con 0x7f1cf01457f0 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.400+0000 7f1ced7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.400+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f1ce8003dc0 con 0x7f1cf01457f0 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.400+0000 7f1ced7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.400+0000 7f1ced7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.400+0000 7f1ced7fa640 10 monclient: dump: 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:27.402 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:27.403 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:27.403 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.400+0000 7f1ceeffd640 4 set_mon_vals no callback set 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.401+0000 7f1cf6f7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.401+0000 7f1cf6f7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.401+0000 7f1cf6f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.401+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd0005080 con 0x7f1cf01457f0 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.401+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f1ce8019d90 con 0x7f1cf01457f0 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.402+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f1ce8083b00 con 0x7f1cf01457f0 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.402+0000 7f1ced7fa640 10 client.4476.objecter ms_dispatch 0x7f1cf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.402+0000 7f1ced7fa640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:27.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.402+0000 7f1ced7fa640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:27.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.402+0000 7f1cf4d17640 10 monclient: get_auth_request con 0x7f1cd406e2b0 auth_method 0 2024-03-29T11:27:27.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.402+0000 7f1ced7fa640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:27:27.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.403+0000 7f1ced7fa640 10 client.4476.objecter ms_handle_connect 0x7f1cd406e2b0 2024-03-29T11:27:27.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.409+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ce804f3e0 con 0x7f1cf01457f0 2024-03-29T11:27:27.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.409+0000 7f1ced7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.406 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.409+0000 7f1ced7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:27.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.799+0000 7f1cf6f7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.799+0000 7f1cf6f7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.799+0000 7f1cf6f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:27.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.799+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cd0005330 con 0x7f1cf01457f0 2024-03-29T11:27:27.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.801+0000 7f1ced7fa640 1 -- 172.21.15.138:0/1043557446 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f1ce8051310 con 0x7f1cf01457f0 2024-03-29T11:27:27.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.801+0000 7f1ced7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.801+0000 7f1ced7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:27.789 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:27.789 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 10 librados: watch_flush enter 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 10 librados: watch_flush exit 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 >> v1:172.21.15.152:6816/3052585572 conn(0x7f1cd406e2b0 legacy=0x7f1cd4070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 10 monclient: shutdown 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 >> v1:172.21.15.138:6789/0 conn(0x7f1cf01457f0 legacy=0x7f1cf0140980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 shutdown_connections 2024-03-29T11:27:27.793 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.805+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 >> 172.21.15.138:0/1043557446 conn(0x7f1cf0066a00 msgr2=0x7f1cf00b5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:27.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.806+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 shutdown_connections 2024-03-29T11:27:27.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.806+0000 7f1cf6f7b640 1 -- 172.21.15.138:0/1043557446 wait complete. 2024-03-29T11:27:27.794 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:27.806+0000 7f1cf6f7b640 1 librados: shutdown 2024-03-29T11:27:27.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:34.617 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:34.767 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.779+0000 7f888e90e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.779+0000 7f888e90e640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.779+0000 7f888e90e640 10 monclient: monmap: 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:34.780474+0000 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:34.780474+0000 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:34.768 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:34.769 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.780+0000 7f888e90e640 1 Processor -- start 2024-03-29T11:27:34.769 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.780+0000 7f888e90e640 1 -- start start 2024-03-29T11:27:34.769 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.780+0000 7f888e90e640 10 monclient: init 2024-03-29T11:27:34.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:34.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 10 monclient(hunting): picked mon.noname-a con 0x7f88881422b0 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:34.771 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 10 monclient(hunting): picked mon.noname-c con 0x7f8888142690 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 10 monclient(hunting): picked mon.noname-b con 0x7f88880b17d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8888142c80 con 0x7f88881422b0 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88880aae40 con 0x7f8888142690 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888813fbd0 con 0x7f88880b17d0 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f888e90e640 10 monclient(hunting): authenticate will time out at 1276.090830s 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f88877fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f8888142690 0x7f88880b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:35084/0 (socket says 172.21.15.138:35084) 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.781+0000 7f88877fe640 1 -- 172.21.15.138:0/2346382499 learned_addr learned my addr 172.21.15.138:0/2346382499 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229287576 0 0) 0x7f888813fbd0 con 0x7f88880b17d0 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.772 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8870003520 con 0x7f88880b17d0 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033714709 0 0) 0x7f8888142c80 con 0x7f88881422b0 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8870003840 con 0x7f88881422b0 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955662263 0 0) 0x7f8870003520 con 0x7f88880b17d0 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 >> v1:172.21.15.152:6789/0 conn(0x7f8888142690 legacy=0x7f88880b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 >> v1:172.21.15.138:6789/0 conn(0x7f88881422b0 legacy=0x7f888813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:27:34.773 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888151940 con 0x7f88880b17d0 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f88800019f0 con 0x7f88880b17d0 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.782+0000 7f8886ffd640 10 monclient: dump: 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:34.774 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:34.775 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:34.775 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:34.775 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:34.775 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f888e90e640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:27:34.775 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f888e90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:34.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8880001b50 con 0x7f88880b17d0 2024-03-29T11:27:34.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f8886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f888e90e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:34.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f888e90e640 4 set_mon_vals no callback set 2024-03-29T11:27:34.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f8886ffd640 1 -- 172.21.15.138:0/2346382499 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8880001ee0 con 0x7f88880b17d0 2024-03-29T11:27:34.780 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f8886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f8886ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.783+0000 7f8886ffd640 10 monclient: dump: 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.784+0000 7f888e90e640 10 monclient: shutdown 2024-03-29T11:27:34.781 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.784+0000 7f888e90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.784+0000 7f888e90e640 1 -- 172.21.15.138:0/2346382499 >> v1:172.21.15.138:6790/0 conn(0x7f88880b17d0 legacy=0x7f88880b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.784+0000 7f888e90e640 1 -- 172.21.15.138:0/2346382499 shutdown_connections 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.784+0000 7f888e90e640 1 -- 172.21.15.138:0/2346382499 >> 172.21.15.138:0/2346382499 conn(0x7f8888066a00 msgr2=0x7f8888066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.785+0000 7f888e90e640 1 -- 172.21.15.138:0/2346382499 shutdown_connections 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.785+0000 7f888e90e640 1 -- 172.21.15.138:0/2346382499 wait complete. 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 10 monclient: monmap: 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:34.782 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 1 librados: starting msgr at 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 1 librados: starting objecter 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 1 Processor -- start 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 1 -- start start 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 1 librados: setting wanted keys 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 1 librados: calling monclient init 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.786+0000 7f888e90e640 10 monclient: init 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888e90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888e90e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888e90e640 10 monclient(hunting): picked mon.noname-b con 0x7f88880b17d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888e90e640 10 monclient(hunting): picked mon.noname-c con 0x7f88881422b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:34.783 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888e90e640 10 monclient(hunting): picked mon.noname-a con 0x7f8888142690 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888ceab640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f8888142690 0x7f8888130860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:56646/0 (socket says 172.21.15.138:56646) 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888ceab640 1 -- 172.21.15.138:0/4043424524 learned_addr learned my addr 172.21.15.138:0/4043424524 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.787+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88880ac7a0 con 0x7f8888142690 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 client.?.objecter ms_handle_connect 0x7f8888142690 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88880aca10 con 0x7f88880b17d0 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888812fc30 con 0x7f88881422b0 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f888e90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f888e90e640 10 monclient(hunting): authenticate will time out at 1276.097167s 2024-03-29T11:27:34.784 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988237591 0 0) 0x7f88880ac7a0 con 0x7f8888142690 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8874004eb0 con 0x7f8888142690 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 client.?.objecter ms_handle_connect 0x7f88881422b0 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 client.?.objecter ms_handle_connect 0x7f88880b17d0 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.788+0000 7f8884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586555109 0 0) 0x7f8874004eb0 con 0x7f8888142690 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88880ac7a0 con 0x7f8888142690 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8880001ee0 con 0x7f8888142690 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:34.785 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740666632 0 0) 0x7f88880aca10 con 0x7f88880b17d0 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.786 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8874004eb0 con 0x7f88880b17d0 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683176017 0 0) 0x7f888812fc30 con 0x7f88881422b0 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88880aca10 con 0x7f88881422b0 2024-03-29T11:27:34.787 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3916878601 0 0) 0x7f88880ac7a0 con 0x7f8888142690 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 >> v1:172.21.15.138:6790/0 conn(0x7f88881422b0 legacy=0x7f8888130150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 >> v1:172.21.15.152:6789/0 conn(0x7f88880b17d0 legacy=0x7f88880ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 monclient: found mon.a 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888132740 con 0x7f8888142690 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f8884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 10 monclient: _renew_subs 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8888132990 con 0x7f8888142690 2024-03-29T11:27:34.788 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 10 monclient: _renew_subs 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.789+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88881e2e70 con 0x7f8888142690 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f888e90e640 1 librados: init done 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8880003c20 con 0x7f8888142690 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f8884ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8880003ff0 con 0x7f8888142690 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f8884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f8884ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f8884ff9640 10 monclient: dump: 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.789 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.790+0000 7f88867fc640 4 set_mon_vals no callback set 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.791+0000 7f888e90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.791+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f8880004150 con 0x7f8888142690 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f888e90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f888e90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:34.790 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8868005080 con 0x7f8888142690 2024-03-29T11:27:34.791 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f8880084ef0 con 0x7f8888142690 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f8884ff9640 10 client.4482.objecter ms_dispatch 0x7f8888000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f8884ff9640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f8884ff9640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.792+0000 7f8884ff9640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.793+0000 7f8887fff640 10 monclient: get_auth_request con 0x7f887406deb0 auth_method 0 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.793+0000 7f8884ff9640 10 client.4482.objecter ms_handle_connect 0x7f887406deb0 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.802+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88800507d0 con 0x7f8888142690 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.802+0000 7f8884ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.792 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:34.802+0000 7f8884ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:35.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.177+0000 7f888e90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:35.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.177+0000 7f888e90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:35.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.177+0000 7f888e90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:35.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.177+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8868005330 con 0x7f8888142690 2024-03-29T11:27:35.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.178+0000 7f8884ff9640 1 -- 172.21.15.138:0/4043424524 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f8880052700 con 0x7f8888142690 2024-03-29T11:27:35.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.178+0000 7f8884ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:35.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.178+0000 7f8884ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:35.167 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:35.168 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:35.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 10 librados: watch_flush enter 2024-03-29T11:27:35.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 10 librados: watch_flush exit 2024-03-29T11:27:35.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:27:35.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:27:35.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 >> v1:172.21.15.152:6816/3052585572 conn(0x7f887406deb0 legacy=0x7f8874070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 10 monclient: shutdown 2024-03-29T11:27:35.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:35.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 >> v1:172.21.15.138:6789/0 conn(0x7f8888142690 legacy=0x7f8888130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 shutdown_connections 2024-03-29T11:27:35.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 >> 172.21.15.138:0/4043424524 conn(0x7f8888066a00 msgr2=0x7f88880b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:35.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 shutdown_connections 2024-03-29T11:27:35.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.182+0000 7f888e90e640 1 -- 172.21.15.138:0/4043424524 wait complete. 2024-03-29T11:27:35.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:35.183+0000 7f888e90e640 1 librados: shutdown 2024-03-29T11:27:35.186 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:41.994 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:42.145 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.156+0000 7fc365354640 10 monclient: get_monmap_and_config 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.156+0000 7fc365354640 10 monclient: build_initial_monmap 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.156+0000 7fc365354640 10 monclient: monmap: 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:42.157299+0000 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:42.157299+0000 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:42.146 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:42.147 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:42.147 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:42.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.156+0000 7fc365354640 1 Processor -- start 2024-03-29T11:27:42.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 1 -- start start 2024-03-29T11:27:42.147 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient: init 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient(hunting): picked mon.noname-a con 0x7fc3600a9ac0 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient(hunting): picked mon.noname-b con 0x7fc36013ee90 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient(hunting): picked mon.noname-c con 0x7fc3600af900 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600afd10 con 0x7fc3600a9ac0 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600af6f0 con 0x7fc3600af900 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600aac80 con 0x7fc36013ee90 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.157+0000 7fc365354640 10 monclient(hunting): authenticate will time out at 1283.466927s 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35effd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc3600a9ac0 0x7fc3600adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:35288/0 (socket says 172.21.15.138:35288) 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35effd640 1 -- 172.21.15.138:0/1142509889 learned_addr learned my addr 172.21.15.138:0/1142509889 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:42.148 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679756384 0 0) 0x7fc3600afd10 con 0x7fc3600a9ac0 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc348003520 con 0x7fc3600a9ac0 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634483295 0 0) 0x7fc348003520 con 0x7fc3600a9ac0 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 >> v1:172.21.15.138:6790/0 conn(0x7fc36013ee90 legacy=0x7fc36013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:42.149 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.158+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 >> v1:172.21.15.152:6789/0 conn(0x7fc3600af900 legacy=0x7fc360149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 1 monclient: found mon.noname-a 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600aaf60 con 0x7fc3600a9ac0 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:42.151 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc365354640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc365354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc350001690 con 0x7fc3600a9ac0 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc35dffb640 10 monclient: dump: 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:42.152 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.161+0000 7fc365354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc350001860 con 0x7fc3600a9ac0 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc35dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc35dffb640 1 -- 172.21.15.138:0/1142509889 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc350001cb0 con 0x7fc3600a9ac0 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc365354640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc365354640 4 set_mon_vals no callback set 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc35dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc35dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.162+0000 7fc35dffb640 10 monclient: dump: 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:42.153 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:42.154 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.163+0000 7fc365354640 10 monclient: shutdown 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.163+0000 7fc365354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.163+0000 7fc365354640 1 -- 172.21.15.138:0/1142509889 >> v1:172.21.15.138:6789/0 conn(0x7fc3600a9ac0 legacy=0x7fc3600adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.164+0000 7fc365354640 1 -- 172.21.15.138:0/1142509889 shutdown_connections 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.164+0000 7fc365354640 1 -- 172.21.15.138:0/1142509889 >> 172.21.15.138:0/1142509889 conn(0x7fc3600b35e0 msgr2=0x7fc3600b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.164+0000 7fc365354640 1 -- 172.21.15.138:0/1142509889 shutdown_connections 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.165+0000 7fc365354640 1 -- 172.21.15.138:0/1142509889 wait complete. 2024-03-29T11:27:42.155 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.166+0000 7fc365354640 10 monclient: build_initial_monmap 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.166+0000 7fc365354640 10 monclient: monmap: 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.166+0000 7fc365354640 1 librados: starting msgr at 2024-03-29T11:27:42.156 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.166+0000 7fc365354640 1 librados: starting objecter 2024-03-29T11:27:42.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.167+0000 7fc365354640 1 Processor -- start 2024-03-29T11:27:42.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.167+0000 7fc365354640 1 -- start start 2024-03-29T11:27:42.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.167+0000 7fc365354640 1 librados: setting wanted keys 2024-03-29T11:27:42.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.167+0000 7fc365354640 1 librados: calling monclient init 2024-03-29T11:27:42.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.167+0000 7fc365354640 10 monclient: init 2024-03-29T11:27:42.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:42.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 10 monclient(hunting): picked mon.noname-c con 0x7fc3600a9ac0 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 10 monclient(hunting): picked mon.noname-b con 0x7fc3600af900 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 10 monclient(hunting): picked mon.noname-a con 0x7fc36013ee90 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360144060 con 0x7fc36013ee90 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360144340 con 0x7fc3600af900 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360143220 con 0x7fc3600a9ac0 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:42.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.168+0000 7fc365354640 10 monclient(hunting): authenticate will time out at 1283.477575s 2024-03-29T11:27:42.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.170+0000 7fc35f7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc36013ee90 0x7fc3601e4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:58690/0 (socket says 172.21.15.138:58690) 2024-03-29T11:27:42.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.170+0000 7fc35f7fe640 1 -- 172.21.15.138:0/2647425205 learned_addr learned my addr 172.21.15.138:0/2647425205 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:42.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.170+0000 7fc3477fe640 10 client.?.objecter ms_handle_connect 0x7fc36013ee90 2024-03-29T11:27:42.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:42.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382492572 0 0) 0x7fc360144060 con 0x7fc36013ee90 2024-03-29T11:27:42.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:27:42.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c004eb0 con 0x7fc36013ee90 2024-03-29T11:27:42.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1994040796 0 0) 0x7fc34c004eb0 con 0x7fc36013ee90 2024-03-29T11:27:42.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc360144060 con 0x7fc36013ee90 2024-03-29T11:27:42.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc358001b70 con 0x7fc36013ee90 2024-03-29T11:27:42.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:42.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): dump: 2024-03-29T11:27:42.161 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:42.162 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.171+0000 7fc3477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:42.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.174+0000 7fc3477fe640 10 client.?.objecter ms_handle_connect 0x7fc3600af900 2024-03-29T11:27:42.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803660636 0 0) 0x7fc360144340 con 0x7fc3600af900 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c004eb0 con 0x7fc3600af900 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1872163592 0 0) 0x7fc360144060 con 0x7fc36013ee90 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 >> v1:172.21.15.138:6790/0 conn(0x7fc3600a9ac0 legacy=0x7fc3601e0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 >> v1:172.21.15.152:6789/0 conn(0x7fc3600af900 legacy=0x7fc3601e3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 monclient: found mon.a 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600721c0 con 0x7fc36013ee90 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 10 monclient: _renew_subs 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3600724c0 con 0x7fc36013ee90 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 10 monclient: _renew_subs 2024-03-29T11:27:42.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:42.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3601e1510 con 0x7fc36013ee90 2024-03-29T11:27:42.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc365354640 1 librados: init done 2024-03-29T11:27:42.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc358001cd0 con 0x7fc36013ee90 2024-03-29T11:27:42.169 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc3477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.175+0000 7fc35cff9640 4 set_mon_vals no callback set 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc358004e20 con 0x7fc36013ee90 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc3477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc3477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc3477fe640 10 monclient: dump: 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:42.170 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc365354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc365354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc365354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.176+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc340005080 con 0x7fc36013ee90 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.178+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fc35801adf0 con 0x7fc36013ee90 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.179+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fc358084e30 con 0x7fc36013ee90 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.179+0000 7fc3477fe640 10 client.4488.objecter ms_dispatch 0x7fc360000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.179+0000 7fc3477fe640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.179+0000 7fc3477fe640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.179+0000 7fc3477fe640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.180+0000 7fc35effd640 10 monclient: get_auth_request con 0x7fc34c06dba0 auth_method 0 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.180+0000 7fc3477fe640 10 client.4488.objecter ms_handle_connect 0x7fc34c06dba0 2024-03-29T11:27:42.177 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.188+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc358050540 con 0x7fc36013ee90 2024-03-29T11:27:42.177 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.188+0000 7fc3477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:42.178 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.188+0000 7fc3477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:42.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.505+0000 7fc365354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.505+0000 7fc365354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.505+0000 7fc365354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:27:42.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.505+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc340005330 con 0x7fc36013ee90 2024-03-29T11:27:42.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.507+0000 7fc3477fe640 1 -- 172.21.15.138:0/2647425205 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fc358052640 con 0x7fc36013ee90 2024-03-29T11:27:42.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.507+0000 7fc3477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.507+0000 7fc3477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:42.497 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:42.498 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:42.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.510+0000 7fc365354640 10 librados: watch_flush enter 2024-03-29T11:27:42.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.510+0000 7fc365354640 10 librados: watch_flush exit 2024-03-29T11:27:42.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.510+0000 7fc365354640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:27:42.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.510+0000 7fc365354640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:27:42.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.511+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 >> v1:172.21.15.152:6816/3052585572 conn(0x7fc34c06dba0 legacy=0x7fc34c070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.511+0000 7fc365354640 10 monclient: shutdown 2024-03-29T11:27:42.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.511+0000 7fc365354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:42.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.511+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 >> v1:172.21.15.138:6789/0 conn(0x7fc36013ee90 legacy=0x7fc3601e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.511+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 shutdown_connections 2024-03-29T11:27:42.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.511+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 >> 172.21.15.138:0/2647425205 conn(0x7fc3600b35e0 msgr2=0x7fc3600abf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:42.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.512+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 shutdown_connections 2024-03-29T11:27:42.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.512+0000 7fc365354640 1 -- 172.21.15.138:0/2647425205 wait complete. 2024-03-29T11:27:42.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:42.512+0000 7fc365354640 1 librados: shutdown 2024-03-29T11:27:42.518 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:49.326 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:49.474 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.484+0000 7f991ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.484+0000 7f991ffff640 10 monclient: build_initial_monmap 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.484+0000 7f991ffff640 10 monclient: monmap: 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:49.486011+0000 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:49.486011+0000 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:49.475 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:49.476 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:49.476 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:49.476 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:49.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.485+0000 7f991ffff640 1 Processor -- start 2024-03-29T11:27:49.477 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.485+0000 7f991ffff640 1 -- start start 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.485+0000 7f991ffff640 10 monclient: init 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f9918006410 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f991802a640 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f99180327d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991802c820 con 0x7f9918006410 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991802a120 con 0x7f99180327d0 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180c75b0 con 0x7f991802a640 2024-03-29T11:27:49.478 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:49.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.486+0000 7f991ffff640 10 monclient(hunting): authenticate will time out at 1290.795949s 2024-03-29T11:27:49.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.487+0000 7f991e7fc640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f991802a640 0x7f99180320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:36050/0 (socket says 172.21.15.138:36050) 2024-03-29T11:27:49.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.487+0000 7f991e7fc640 1 -- 172.21.15.138:0/80612767 learned_addr learned my addr 172.21.15.138:0/80612767 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.491+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186706948 0 0) 0x7f99180c75b0 con 0x7f991802a640 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.491+0000 7f991dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.491+0000 7f991dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.491+0000 7f991dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.491+0000 7f991dffb640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.491+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9908003520 con 0x7f991802a640 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.493+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007588186 0 0) 0x7f9908003520 con 0x7f991802a640 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.493+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 >> v1:172.21.15.152:6789/0 conn(0x7f99180327d0 legacy=0x7f99180d1550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 >> v1:172.21.15.138:6789/0 conn(0x7f9918006410 legacy=0x7f99180c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:49.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 1 monclient: found mon.noname-b 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918033110 con 0x7f991802a640 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9914001670 con 0x7f991802a640 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:49.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: dump: 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991ffff640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99140018d0 con 0x7f991802a640 2024-03-29T11:27:49.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.494+0000 7f991ffff640 4 set_mon_vals no callback set 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.496+0000 7f991dffb640 1 -- 172.21.15.138:0/80612767 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9914001d20 con 0x7f991802a640 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.496+0000 7f991dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.496+0000 7f991dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.496+0000 7f991dffb640 10 monclient: dump: 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:49.490 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.496+0000 7f991ffff640 10 monclient: shutdown 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.496+0000 7f991ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.496+0000 7f991ffff640 1 -- 172.21.15.138:0/80612767 >> v1:172.21.15.138:6790/0 conn(0x7f991802a640 legacy=0x7f99180320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.497+0000 7f991ffff640 1 -- 172.21.15.138:0/80612767 shutdown_connections 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.497+0000 7f991ffff640 1 -- 172.21.15.138:0/80612767 >> 172.21.15.138:0/80612767 conn(0x7f99180058f0 msgr2=0x7f9918005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:49.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.497+0000 7f991ffff640 1 -- 172.21.15.138:0/80612767 shutdown_connections 2024-03-29T11:27:49.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.497+0000 7f991ffff640 1 -- 172.21.15.138:0/80612767 wait complete. 2024-03-29T11:27:49.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.498+0000 7f991ffff640 10 monclient: build_initial_monmap 2024-03-29T11:27:49.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.498+0000 7f991ffff640 10 monclient: monmap: 2024-03-29T11:27:49.494 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:49.494 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:49.494 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.498+0000 7f991ffff640 1 librados: starting msgr at 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.498+0000 7f991ffff640 1 librados: starting objecter 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.498+0000 7f991ffff640 1 Processor -- start 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 1 -- start start 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 1 librados: setting wanted keys 2024-03-29T11:27:49.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 1 librados: calling monclient init 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient: init 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f9918006410 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f991802a640 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f99180327d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180be170 con 0x7f9918006410 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991802d7d0 con 0x7f99180327d0 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9918031b90 con 0x7f991802a640 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.499+0000 7f991ffff640 10 monclient(hunting): authenticate will time out at 1290.808951s 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f991e7fc640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f991802a640 0x7f9918160320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:35996/0 (socket says 172.21.15.138:35996) 2024-03-29T11:27:49.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f991e7fc640 1 -- 172.21.15.138:0/485363347 learned_addr learned my addr 172.21.15.138:0/485363347 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 10 client.?.objecter ms_handle_connect 0x7f991802a640 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731403945 0 0) 0x7f9918031b90 con 0x7f991802a640 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990c004eb0 con 0x7f991802a640 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.503+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1937642656 0 0) 0x7f990c004eb0 con 0x7f991802a640 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9918031b90 con 0x7f991802a640 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9914001d20 con 0x7f991802a640 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 10 monclient(hunting): dump: 2024-03-29T11:27:49.497 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4028868683 0 0) 0x7f9918031b90 con 0x7f991802a640 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 >> v1:172.21.15.152:6789/0 conn(0x7f99180327d0 legacy=0x7f9918169ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:49.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 >> v1:172.21.15.138:6789/0 conn(0x7f9918006410 legacy=0x7f991802cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 1 monclient: found mon.c 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.504+0000 7f99077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918031d20 con 0x7f991802a640 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9914001530 con 0x7f991802a640 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f991d7fa640 4 set_mon_vals no callback set 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9914005170 con 0x7f991802a640 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f99077fe640 10 monclient: dump: 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:49.500 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f991ffff640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f991ffff640 10 monclient: _renew_subs 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f991ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9918160a30 con 0x7f991802a640 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.505+0000 7f991ffff640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.506+0000 7f991ffff640 10 monclient: _renew_subs 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.506+0000 7f991ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:49.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.506+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f991816a1c0 con 0x7f991802a640 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.506+0000 7f991ffff640 1 librados: init done 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.508+0000 7f991ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.508+0000 7f991ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.508+0000 7f991ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.508+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9918000f50 con 0x7f991802a640 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.509+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f991401b140 con 0x7f991802a640 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.510+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f9914084e00 con 0x7f991802a640 2024-03-29T11:27:49.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.510+0000 7f99077fe640 10 client.4550.objecter ms_dispatch 0x7f9920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:49.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.510+0000 7f99077fe640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:49.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.510+0000 7f99077fe640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:49.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.510+0000 7f99077fe640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:27:49.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.511+0000 7f991effd640 10 monclient: get_auth_request con 0x7f990c06dc10 auth_method 0 2024-03-29T11:27:49.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.511+0000 7f99077fe640 10 client.4550.objecter ms_handle_connect 0x7f990c06dc10 2024-03-29T11:27:49.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.518+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99140506e0 con 0x7f991802a640 2024-03-29T11:27:49.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.518+0000 7f99077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.518+0000 7f99077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:49.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.845+0000 7f991ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.845+0000 7f991ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.846+0000 7f991ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:49.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.846+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9918160bc0 con 0x7f991802a640 2024-03-29T11:27:49.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.849+0000 7f99077fe640 1 -- 172.21.15.138:0/485363347 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f9914052610 con 0x7f991802a640 2024-03-29T11:27:49.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.849+0000 7f99077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.849+0000 7f99077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:49.838 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:49.838 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:49.841 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.852+0000 7f991ffff640 10 librados: watch_flush enter 2024-03-29T11:27:49.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.852+0000 7f991ffff640 10 librados: watch_flush exit 2024-03-29T11:27:49.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.852+0000 7f991ffff640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:27:49.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.852+0000 7f991ffff640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:27:49.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.853+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 >> v1:172.21.15.152:6816/3052585572 conn(0x7f990c06dc10 legacy=0x7f990c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.853+0000 7f991ffff640 10 monclient: shutdown 2024-03-29T11:27:49.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.853+0000 7f991ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:49.842 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.853+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 >> v1:172.21.15.138:6790/0 conn(0x7f991802a640 legacy=0x7f9918160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.853+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 shutdown_connections 2024-03-29T11:27:49.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.853+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 >> 172.21.15.138:0/485363347 conn(0x7f99180058f0 msgr2=0x7f9918036a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:49.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.854+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 shutdown_connections 2024-03-29T11:27:49.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.854+0000 7f991ffff640 1 -- 172.21.15.138:0/485363347 wait complete. 2024-03-29T11:27:49.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:49.854+0000 7f991ffff640 1 librados: shutdown 2024-03-29T11:27:49.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:56.670 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:56.821 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.832+0000 7f00aa562640 10 monclient: get_monmap_and_config 2024-03-29T11:27:56.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.832+0000 7f00aa562640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.822 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.832+0000 7f00aa562640 10 monclient: monmap: 2024-03-29T11:27:56.822 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:56.822 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:56.822 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:27:56.833629+0000 2024-03-29T11:27:56.822 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:27:56.833629+0000 2024-03-29T11:27:56.823 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.823 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:56.823 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:56.823 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:27:56.823 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:27:56.823 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.834+0000 7f00aa562640 1 Processor -- start 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.835+0000 7f00aa562640 1 -- start start 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.835+0000 7f00aa562640 10 monclient: init 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.835+0000 7f00aa562640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.835+0000 7f00aa562640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.835+0000 7f00aa562640 10 monclient(hunting): picked mon.noname-b con 0x7f009c036a60 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00aa562640 10 monclient(hunting): picked mon.noname-a con 0x7f009c0c5de0 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:56.831 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00aa562640 10 monclient(hunting): picked mon.noname-c con 0x7f009c0c3070 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00aa562640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c02cc60 con 0x7f009c0c5de0 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00aa562640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c02e040 con 0x7f009c0c3070 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00aa562640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c02e320 con 0x7f009c036a60 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00aa562640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00aa562640 10 monclient(hunting): authenticate will time out at 1298.145106s 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00a9560640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f009c036a60 0x7f009c0c56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:45218/0 (socket says 172.21.15.138:45218) 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f00a9560640 1 -- 172.21.15.138:0/1922175766 learned_addr learned my addr 172.21.15.138:0/1922175766 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833963159 0 0) 0x7f009c02e320 con 0x7f009c036a60 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f009bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f009bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f009bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f009bfff640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.836+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0090003520 con 0x7f009c036a60 2024-03-29T11:27:56.832 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.837+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864916632 0 0) 0x7f0090003520 con 0x7f009c036a60 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.837+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 >> v1:172.21.15.152:6789/0 conn(0x7f009c0c3070 legacy=0x7f009c0d0fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.837+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 >> v1:172.21.15.138:6789/0 conn(0x7f009c0c5de0 legacy=0x7f009c0c2960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 1 monclient: found mon.noname-b 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c02c830 con 0x7f009c036a60 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f00a404f730 con 0x7f009c036a60 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.833 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f009bfff640 10 monclient: dump: 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:56.834 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f00aa562640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.839+0000 7f00aa562640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a40008d0 con 0x7f009c036a60 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f009bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f009bfff640 1 -- 172.21.15.138:0/1922175766 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f00a40758b0 con 0x7f009c036a60 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f009bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f009bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f009bfff640 10 monclient: dump: 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:56.835 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f00aa562640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.840+0000 7f00aa562640 4 set_mon_vals no callback set 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.841+0000 7f00aa562640 10 monclient: shutdown 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.841+0000 7f00aa562640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:56.836 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.841+0000 7f00aa562640 1 -- 172.21.15.138:0/1922175766 >> v1:172.21.15.138:6790/0 conn(0x7f009c036a60 legacy=0x7f009c0c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.841+0000 7f00aa562640 1 -- 172.21.15.138:0/1922175766 shutdown_connections 2024-03-29T11:27:56.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.841+0000 7f00aa562640 1 -- 172.21.15.138:0/1922175766 >> 172.21.15.138:0/1922175766 conn(0x7f009c005c00 msgr2=0x7f009c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:56.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.843+0000 7f00aa562640 1 -- 172.21.15.138:0/1922175766 shutdown_connections 2024-03-29T11:27:56.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.843+0000 7f00aa562640 1 -- 172.21.15.138:0/1922175766 wait complete. 2024-03-29T11:27:56.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.837 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 10 monclient: monmap: 2024-03-29T11:27:56.837 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 1 librados: starting msgr at 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 1 librados: starting objecter 2024-03-29T11:27:56.838 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 1 Processor -- start 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 1 -- start start 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 1 librados: setting wanted keys 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 1 librados: calling monclient init 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.844+0000 7f00aa562640 10 monclient: init 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 10 monclient(hunting): picked mon.noname-b con 0x7f009c036a60 addr v1:172.21.15.152:6789/0 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 10 monclient(hunting): picked mon.noname-c con 0x7f009c0c3070 addr v1:172.21.15.138:6790/0 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 10 monclient(hunting): picked mon.noname-a con 0x7f009c0c7140 addr v1:172.21.15.138:6789/0 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0c7ac0 con 0x7f009c0c7140 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0c9250 con 0x7f009c036a60 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0c9530 con 0x7f009c0c3070 2024-03-29T11:27:56.839 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.840 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.845+0000 7f00aa562640 10 monclient(hunting): authenticate will time out at 1298.154690s 2024-03-29T11:27:56.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f00a8d5f640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f009c0c3070 0x7f009c0bdbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:46290/0 (socket says 172.21.15.138:46290) 2024-03-29T11:27:56.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f00a8d5f640 1 -- 172.21.15.138:0/3018160038 learned_addr learned my addr 172.21.15.138:0/3018160038 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:27:56.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 10 client.?.objecter ms_handle_connect 0x7f009c0c3070 2024-03-29T11:27:56.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.843 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367421904 0 0) 0x7f009c0c9530 con 0x7f009c0c3070 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.849+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0088004eb0 con 0x7f009c0c3070 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 client.?.objecter ms_handle_connect 0x7f009c0c7140 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 client.?.objecter ms_handle_connect 0x7f009c036a60 2024-03-29T11:27:56.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2689865066 0 0) 0x7f0088004eb0 con 0x7f009c0c3070 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0088005300 con 0x7f009c0c3070 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f00a0001790 con 0x7f009c0c3070 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:56.845 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:56.846 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.846 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.846 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.846 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:56.846 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:56.846 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:56.846 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:56.847 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:56.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:56.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882715007 0 0) 0x7f009c0c7ac0 con 0x7f009c0c7140 2024-03-29T11:27:56.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:27:56.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0088004eb0 con 0x7f009c0c7140 2024-03-29T11:27:56.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067898587 0 0) 0x7f009c0c9250 con 0x7f009c036a60 2024-03-29T11:27:56.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:27:56.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00880059e0 con 0x7f009c036a60 2024-03-29T11:27:56.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2024219693 0 0) 0x7f0088005300 con 0x7f009c0c3070 2024-03-29T11:27:56.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 >> v1:172.21.15.152:6789/0 conn(0x7f009c036a60 legacy=0x7f009c0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 >> v1:172.21.15.138:6789/0 conn(0x7f009c0c7140 legacy=0x7f009c0ba840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 monclient: found mon.c 2024-03-29T11:27:56.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c0bfe50 con 0x7f009c0c3070 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.850+0000 7f0099ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.851+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a0001d20 con 0x7f009c0c3070 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.851+0000 7f0099ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.851+0000 7f009affd640 4 set_mon_vals no callback set 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.851+0000 7f00aa562640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.851+0000 7f00aa562640 10 monclient: _renew_subs 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.851+0000 7f00aa562640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.851+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c0c9950 con 0x7f009c0c3070 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f00aa562640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f00aa562640 10 monclient: _renew_subs 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f00aa562640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:56.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c0baf50 con 0x7f009c0c3070 2024-03-29T11:27:56.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f00aa562640 1 librados: init done 2024-03-29T11:27:56.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f00a0004a00 con 0x7f009c0c3070 2024-03-29T11:27:56.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f0099ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f0099ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:56.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.852+0000 7f0099ffb640 10 monclient: dump: 2024-03-29T11:27:56.852 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:27:56.852 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:27:56.853 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.853 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:27:56.853 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.853 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:27:56.853 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:27:56.853 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:27:56.853 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:27:56.854 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:27:56.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.853+0000 7f00aa562640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.854+0000 7f00aa562640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.854+0000 7f00aa562640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:56.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.854+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009c005120 con 0x7f009c0c3070 2024-03-29T11:27:56.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.854+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f00a001aa50 con 0x7f009c0c3070 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.855+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f00a00545c0 con 0x7f009c0c3070 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.855+0000 7f0099ffb640 10 client.4556.objecter ms_dispatch 0x7f00a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.855+0000 7f0099ffb640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.855+0000 7f0099ffb640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.855+0000 7f0099ffb640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.855+0000 7f00a9560640 10 monclient: get_auth_request con 0x7f008806e3b0 auth_method 0 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.857+0000 7f0099ffb640 10 client.4556.objecter ms_handle_connect 0x7f008806e3b0 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.863+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00a0050860 con 0x7f009c0c3070 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.863+0000 7f0099ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:56.863+0000 7f0099ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:57.177 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.188+0000 7f00aa562640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:57.178 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.188+0000 7f00aa562640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:57.178 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.188+0000 7f00aa562640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:27:57.178 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.188+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f009c0c77d0 con 0x7f009c0c3070 2024-03-29T11:27:57.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.190+0000 7f0099ffb640 1 -- 172.21.15.138:0/3018160038 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f00a0050860 con 0x7f009c0c3070 2024-03-29T11:27:57.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.190+0000 7f0099ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:57.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.190+0000 7f0099ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:57.179 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:27:57.180 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:57.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.194+0000 7f00aa562640 10 librados: watch_flush enter 2024-03-29T11:27:57.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.194+0000 7f00aa562640 10 librados: watch_flush exit 2024-03-29T11:27:57.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.194+0000 7f00aa562640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:27:57.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.194+0000 7f00aa562640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:27:57.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 >> v1:172.21.15.152:6816/3052585572 conn(0x7f008806e3b0 legacy=0x7f0088070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 10 monclient: shutdown 2024-03-29T11:27:57.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:57.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 >> v1:172.21.15.138:6790/0 conn(0x7f009c0c3070 legacy=0x7f009c0bdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 shutdown_connections 2024-03-29T11:27:57.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 >> 172.21.15.138:0/3018160038 conn(0x7f009c005c00 msgr2=0x7f009c0c1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:57.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 shutdown_connections 2024-03-29T11:27:57.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 1 -- 172.21.15.138:0/3018160038 wait complete. 2024-03-29T11:27:57.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:27:57.195+0000 7f00aa562640 1 librados: shutdown 2024-03-29T11:27:57.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:04.011 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:04.157 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.166+0000 7f655b5d6640 10 monclient: get_monmap_and_config 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.166+0000 7f655b5d6640 10 monclient: build_initial_monmap 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.166+0000 7f655b5d6640 10 monclient: monmap: 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:04.168075+0000 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:04.168075+0000 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:04.158 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.167+0000 7f655b5d6640 1 Processor -- start 2024-03-29T11:28:04.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.168+0000 7f655b5d6640 1 -- start start 2024-03-29T11:28:04.159 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.168+0000 7f655b5d6640 10 monclient: init 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f65540a9ac0 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f655413ee90 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f65540af900 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540afd10 con 0x7f655413ee90 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540af6f0 con 0x7f65540af900 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540aac80 con 0x7f65540a9ac0 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:04.160 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.169+0000 7f655b5d6640 10 monclient(hunting): authenticate will time out at 1305.478217s 2024-03-29T11:28:04.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f6559372640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f65540a9ac0 0x7f65540adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:60790/0 (socket says 172.21.15.138:60790) 2024-03-29T11:28:04.161 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f6559372640 1 -- 172.21.15.138:0/2985640810 learned_addr learned my addr 172.21.15.138:0/2985640810 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357196896 0 0) 0x7f65540afd10 con 0x7f655413ee90 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6540003520 con 0x7f655413ee90 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219949442 0 0) 0x7f65540aac80 con 0x7f65540a9ac0 2024-03-29T11:28:04.163 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:28:04.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.172+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65540afd10 con 0x7f65540a9ac0 2024-03-29T11:28:04.164 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173989535 0 0) 0x7f6540003520 con 0x7f655413ee90 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 >> v1:172.21.15.138:6790/0 conn(0x7f65540a9ac0 legacy=0x7f65540adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 >> v1:172.21.15.152:6789/0 conn(0x7f65540af900 legacy=0x7f6554149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 monclient: found mon.noname-a 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65540aaf60 con 0x7f655413ee90 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: _finish_auth 0 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:04.165 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f654c001a10 con 0x7f655413ee90 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: dump: 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:04.166 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f655b5d6640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f655b5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f654c001b70 con 0x7f655413ee90 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 1 -- 172.21.15.138:0/2985640810 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f654c001f80 con 0x7f655413ee90 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f654bfff640 10 monclient: dump: 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:04.167 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:04.168 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:04.168 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:04.168 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:04.168 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f655b5d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:04.168 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.173+0000 7f655b5d6640 4 set_mon_vals no callback set 2024-03-29T11:28:04.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.174+0000 7f655b5d6640 10 monclient: shutdown 2024-03-29T11:28:04.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.174+0000 7f655b5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:04.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.174+0000 7f655b5d6640 1 -- 172.21.15.138:0/2985640810 >> v1:172.21.15.138:6789/0 conn(0x7f655413ee90 legacy=0x7f655413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.175+0000 7f655b5d6640 1 -- 172.21.15.138:0/2985640810 shutdown_connections 2024-03-29T11:28:04.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.175+0000 7f655b5d6640 1 -- 172.21.15.138:0/2985640810 >> 172.21.15.138:0/2985640810 conn(0x7f65540b35e0 msgr2=0x7f65540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:04.170 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.175+0000 7f655b5d6640 1 -- 172.21.15.138:0/2985640810 shutdown_connections 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.175+0000 7f655b5d6640 1 -- 172.21.15.138:0/2985640810 wait complete. 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.176+0000 7f655b5d6640 10 monclient: build_initial_monmap 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.176+0000 7f655b5d6640 10 monclient: monmap: 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.176+0000 7f655b5d6640 1 librados: starting msgr at 2024-03-29T11:28:04.171 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.176+0000 7f655b5d6640 1 librados: starting objecter 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.176+0000 7f655b5d6640 1 Processor -- start 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.177+0000 7f655b5d6640 1 -- start start 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.177+0000 7f655b5d6640 1 librados: setting wanted keys 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.177+0000 7f655b5d6640 1 librados: calling monclient init 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.177+0000 7f655b5d6640 10 monclient: init 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.178+0000 7f655b5d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.178+0000 7f655b5d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.178+0000 7f655b5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f65540a9ac0 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f655b5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f65540af900 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f655b5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f655413ee90 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f655b5d6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554141960 con 0x7f655413ee90 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f655b5d6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655413f950 con 0x7f65540a9ac0 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f655b5d6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554140790 con 0x7f65540af900 2024-03-29T11:28:04.172 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f655b5d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:04.173 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f655b5d6640 10 monclient(hunting): authenticate will time out at 1305.489361s 2024-03-29T11:28:04.173 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f6558b71640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f65540af900 0x7f6554126860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:34774/0 (socket says 172.21.15.138:34774) 2024-03-29T11:28:04.173 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.180+0000 7f6558b71640 1 -- 172.21.15.138:0/2381569746 learned_addr learned my addr 172.21.15.138:0/2381569746 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:04.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 client.?.objecter ms_handle_connect 0x7f65540af900 2024-03-29T11:28:04.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:04.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 client.?.objecter ms_handle_connect 0x7f65540a9ac0 2024-03-29T11:28:04.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:04.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266647514 0 0) 0x7f6554140790 con 0x7f65540af900 2024-03-29T11:28:04.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.179 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.184+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6538004eb0 con 0x7f65540af900 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670042620 0 0) 0x7f655413f950 con 0x7f65540a9ac0 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6554140790 con 0x7f65540a9ac0 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485003170 0 0) 0x7f6538004eb0 con 0x7f65540af900 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f655413f950 con 0x7f65540af900 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 client.?.objecter ms_handle_connect 0x7f655413ee90 2024-03-29T11:28:04.180 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f654c001fc0 con 0x7f65540af900 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.185+0000 7f6549ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:04.181 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769976480 0 0) 0x7f6554140790 con 0x7f65540a9ac0 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6538004eb0 con 0x7f65540a9ac0 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f65440018b0 con 0x7f65540a9ac0 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:04.182 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178242694 0 0) 0x7f6554141960 con 0x7f655413ee90 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6554140790 con 0x7f655413ee90 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2208343951 0 0) 0x7f6538004eb0 con 0x7f65540a9ac0 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.186+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 >> v1:172.21.15.138:6790/0 conn(0x7f65540af900 legacy=0x7f6554126860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 >> v1:172.21.15.138:6789/0 conn(0x7f655413ee90 legacy=0x7f65541d55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 1 monclient: found mon.b 2024-03-29T11:28:04.183 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554141bb0 con 0x7f65540a9ac0 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f6549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f655b5d6640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f655b5d6640 10 monclient: _renew_subs 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f655b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.187+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6554126f70 con 0x7f65540a9ac0 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f655b5d6640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f655b5d6640 10 monclient: _renew_subs 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f655b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6554142490 con 0x7f65540a9ac0 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f655b5d6640 1 librados: init done 2024-03-29T11:28:04.184 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6544001d20 con 0x7f65540a9ac0 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f6549ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f6544004980 con 0x7f65540a9ac0 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f6549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f6549ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f6549ffb640 10 monclient: dump: 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:04.185 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:04.186 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:04.186 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.188+0000 7f654b7fe640 4 set_mon_vals no callback set 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.190+0000 7f655b5d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.190+0000 7f655b5d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.190+0000 7f655b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.190+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6534005080 con 0x7f65540a9ac0 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.191+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f6544004ae0 con 0x7f65540a9ac0 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.192+0000 7f6558b71640 10 monclient: get_auth_request con 0x7f653806dfc0 auth_method 0 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.192+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f6544084820 con 0x7f65540a9ac0 2024-03-29T11:28:04.187 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.192+0000 7f6549ffb640 10 client.4453.objecter ms_dispatch 0x7f6554000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:04.188 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.192+0000 7f6549ffb640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:04.188 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.192+0000 7f6549ffb640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:04.188 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.192+0000 7f6549ffb640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:28:04.188 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.192+0000 7f6549ffb640 10 client.4453.objecter ms_handle_connect 0x7f653806dfc0 2024-03-29T11:28:04.189 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.200+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6544050100 con 0x7f65540a9ac0 2024-03-29T11:28:04.190 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.200+0000 7f6549ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:04.190 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.200+0000 7f6549ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:04.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.534+0000 7f655b5d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.534+0000 7f655b5d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.534+0000 7f655b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:04.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.534+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6534005330 con 0x7f65540a9ac0 2024-03-29T11:28:04.526 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:04.526 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:04.527 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.536+0000 7f6549ffb640 1 -- 172.21.15.138:0/2381569746 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f6544052030 con 0x7f65540a9ac0 2024-03-29T11:28:04.527 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.536+0000 7f6549ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.527 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.536+0000 7f6549ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:04.530 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 10 librados: watch_flush enter 2024-03-29T11:28:04.530 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 10 librados: watch_flush exit 2024-03-29T11:28:04.530 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:28:04.530 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:28:04.530 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 >> v1:172.21.15.152:6816/3052585572 conn(0x7f653806dfc0 legacy=0x7f6538070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.531 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 10 monclient: shutdown 2024-03-29T11:28:04.531 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:04.531 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.540+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 >> v1:172.21.15.152:6789/0 conn(0x7f65540a9ac0 legacy=0x7f6554140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.533 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.544+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 shutdown_connections 2024-03-29T11:28:04.533 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.544+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 >> 172.21.15.138:0/2381569746 conn(0x7f65540b35e0 msgr2=0x7f65540ad1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:04.533 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.544+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 shutdown_connections 2024-03-29T11:28:04.534 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.544+0000 7f655b5d6640 1 -- 172.21.15.138:0/2381569746 wait complete. 2024-03-29T11:28:04.534 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:04.544+0000 7f655b5d6640 1 librados: shutdown 2024-03-29T11:28:04.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:11.357 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.520+0000 7f9b97fff640 10 monclient: get_monmap_and_config 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.520+0000 7f9b97fff640 10 monclient: build_initial_monmap 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.520+0000 7f9b97fff640 10 monclient: monmap: 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:11.521133+0000 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:11.521133+0000 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:11.510 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:11.511 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:11.511 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:11.511 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:11.511 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.521+0000 7f9b97fff640 1 Processor -- start 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 1 -- start start 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient: init 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient(hunting): picked mon.noname-c con 0x7f9b980b2800 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient(hunting): picked mon.noname-a con 0x7f9b980b3220 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient(hunting): picked mon.noname-b con 0x7f9b981439c0 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b980b3840 con 0x7f9b980b3220 2024-03-29T11:28:11.516 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9806f230 con 0x7f9b980b2800 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9806ef50 con 0x7f9b981439c0 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.522+0000 7f9b97fff640 10 monclient(hunting): authenticate will time out at 1312.831030s 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.524+0000 7f9b977fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f9b981439c0 0x7f9b98151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:57472/0 (socket says 172.21.15.138:57472) 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.524+0000 7f9b977fe640 1 -- 172.21.15.138:0/3151781750 learned_addr learned my addr 172.21.15.138:0/3151781750 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506704243 0 0) 0x7f9b9806f230 con 0x7f9b980b2800 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b84003520 con 0x7f9b980b2800 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964428776 0 0) 0x7f9b84003520 con 0x7f9b980b2800 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 >> v1:172.21.15.138:6790/0 conn(0x7f9b981439c0 legacy=0x7f9b98151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:11.517 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.526+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 >> v1:172.21.15.138:6789/0 conn(0x7f9b980b3220 legacy=0x7f9b98146710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:11.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:11.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 1 monclient: found mon.noname-c 2024-03-29T11:28:11.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:11.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:28:11.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b98071d30 con 0x7f9b980b2800 2024-03-29T11:28:11.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:11.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:11.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:11.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9b88001a10 con 0x7f9b980b2800 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: dump: 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:11.520 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b97fff640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b97fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b88001be0 con 0x7f9b980b2800 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 1 -- 172.21.15.138:0/3151781750 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9b88001ff0 con 0x7f9b980b2800 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:11.521 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b95ffb640 10 monclient: dump: 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:11.522 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b97fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:11.523 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.529+0000 7f9b97fff640 4 set_mon_vals no callback set 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.530+0000 7f9b97fff640 10 monclient: shutdown 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.530+0000 7f9b97fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.530+0000 7f9b97fff640 1 -- 172.21.15.138:0/3151781750 >> v1:172.21.15.152:6789/0 conn(0x7f9b980b2800 legacy=0x7f9b981432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.530+0000 7f9b97fff640 1 -- 172.21.15.138:0/3151781750 shutdown_connections 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.530+0000 7f9b97fff640 1 -- 172.21.15.138:0/3151781750 >> 172.21.15.138:0/3151781750 conn(0x7f9b98066a00 msgr2=0x7f9b98066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.533+0000 7f9b97fff640 1 -- 172.21.15.138:0/3151781750 shutdown_connections 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.533+0000 7f9b97fff640 1 -- 172.21.15.138:0/3151781750 wait complete. 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.534+0000 7f9b97fff640 10 monclient: build_initial_monmap 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.534+0000 7f9b97fff640 10 monclient: monmap: 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:11.524 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.534+0000 7f9b97fff640 1 librados: starting msgr at 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.534+0000 7f9b97fff640 1 librados: starting objecter 2024-03-29T11:28:11.525 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.535+0000 7f9b97fff640 1 Processor -- start 2024-03-29T11:28:11.535 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.535+0000 7f9b97fff640 1 -- start start 2024-03-29T11:28:11.535 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.535+0000 7f9b97fff640 1 librados: setting wanted keys 2024-03-29T11:28:11.535 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.535+0000 7f9b97fff640 1 librados: calling monclient init 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.535+0000 7f9b97fff640 10 monclient: init 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 10 monclient(hunting): picked mon.noname-a con 0x7f9b980b2800 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 10 monclient(hunting): picked mon.noname-c con 0x7f9b980b3220 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 10 monclient(hunting): picked mon.noname-b con 0x7f9b981439c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b981d2c40 con 0x7f9b980b2800 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b981d3ac0 con 0x7f9b981439c0 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9813ce30 con 0x7f9b980b3220 2024-03-29T11:28:11.536 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.536+0000 7f9b97fff640 10 monclient(hunting): authenticate will time out at 1312.844181s 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.539+0000 7f9b967fc640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f9b980b3220 0x7f9b981d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:59468/0 (socket says 172.21.15.138:59468) 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.539+0000 7f9b967fc640 1 -- 172.21.15.138:0/827891728 learned_addr learned my addr 172.21.15.138:0/827891728 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 client.?.objecter ms_handle_connect 0x7f9b980b3220 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101171173 0 0) 0x7f9b9813ce30 con 0x7f9b980b3220 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b78004eb0 con 0x7f9b980b3220 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3420762225 0 0) 0x7f9b78004eb0 con 0x7f9b980b3220 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b9813ce30 con 0x7f9b980b3220 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9b80004050 con 0x7f9b980b3220 2024-03-29T11:28:11.537 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): dump: 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:11.538 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2868870513 0 0) 0x7f9b9813ce30 con 0x7f9b980b3220 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 >> v1:172.21.15.152:6789/0 conn(0x7f9b981439c0 legacy=0x7f9b9813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 >> v1:172.21.15.138:6789/0 conn(0x7f9b980b2800 legacy=0x7f9b981d33b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.541+0000 7f9b8ffff640 1 monclient: found mon.c 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b981d2e90 con 0x7f9b980b3220 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 10 monclient: _renew_subs 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b981d07b0 con 0x7f9b980b3220 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:11.539 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 10 monclient: _renew_subs 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b981cf8b0 con 0x7f9b980b3220 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b97fff640 1 librados: init done 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b80000ed0 con 0x7f9b980b3220 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9b800012e0 con 0x7f9b980b3220 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b8ffff640 10 monclient: dump: 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.540 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.542+0000 7f9b94ff9640 4 set_mon_vals no callback set 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.543+0000 7f9b97fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.543+0000 7f9b97fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.543+0000 7f9b97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.543+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7c005080 con 0x7f9b980b3220 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.544+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f9b80001440 con 0x7f9b980b3220 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.545+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f9b80084ab0 con 0x7f9b980b3220 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.545+0000 7f9b8ffff640 10 client.4568.objecter ms_dispatch 0x7f9b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.545+0000 7f9b8ffff640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:11.541 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.545+0000 7f9b8ffff640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:11.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.545+0000 7f9b8ffff640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:28:11.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.545+0000 7f9b96ffd640 10 monclient: get_auth_request con 0x7f9b7806dc90 auth_method 0 2024-03-29T11:28:11.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.546+0000 7f9b8ffff640 10 client.4568.objecter ms_handle_connect 0x7f9b7806dc90 2024-03-29T11:28:11.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.552+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b800503f0 con 0x7f9b980b3220 2024-03-29T11:28:11.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.552+0000 7f9b8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.542 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.552+0000 7f9b8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:11.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.879+0000 7f9b97fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.879+0000 7f9b97fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.879+0000 7f9b97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:11.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.879+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b7c005330 con 0x7f9b980b3220 2024-03-29T11:28:11.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.881+0000 7f9b8ffff640 1 -- 172.21.15.138:0/827891728 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f9b80084160 con 0x7f9b980b3220 2024-03-29T11:28:11.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.881+0000 7f9b8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.881+0000 7f9b8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:11.872 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:11.872 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:11.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.885+0000 7f9b97fff640 10 librados: watch_flush enter 2024-03-29T11:28:11.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.885+0000 7f9b97fff640 10 librados: watch_flush exit 2024-03-29T11:28:11.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.885+0000 7f9b97fff640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:28:11.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.885+0000 7f9b97fff640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:28:11.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.886+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 >> v1:172.21.15.152:6816/3052585572 conn(0x7f9b7806dc90 legacy=0x7f9b78070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.886+0000 7f9b97fff640 10 monclient: shutdown 2024-03-29T11:28:11.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.886+0000 7f9b97fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:11.875 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.886+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 >> v1:172.21.15.138:6790/0 conn(0x7f9b980b3220 legacy=0x7f9b981d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.887+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 shutdown_connections 2024-03-29T11:28:11.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.887+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 >> 172.21.15.138:0/827891728 conn(0x7f9b98066a00 msgr2=0x7f9b980b7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:11.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.888+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 shutdown_connections 2024-03-29T11:28:11.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.888+0000 7f9b97fff640 1 -- 172.21.15.138:0/827891728 wait complete. 2024-03-29T11:28:11.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:11.889+0000 7f9b97fff640 1 librados: shutdown 2024-03-29T11:28:11.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:18.699 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:18.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.857+0000 7f87b64fb640 10 monclient: get_monmap_and_config 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.857+0000 7f87b64fb640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.857+0000 7f87b64fb640 10 monclient: monmap: 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:18.858084+0000 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:18.858084+0000 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:18.847 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:18.848 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:18.848 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.859+0000 7f87b64fb640 1 Processor -- start 2024-03-29T11:28:18.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.859+0000 7f87b64fb640 1 -- start start 2024-03-29T11:28:18.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.859+0000 7f87b64fb640 10 monclient: init 2024-03-29T11:28:18.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.860+0000 7f87b64fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.860+0000 7f87b64fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:18.851 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.860+0000 7f87b64fb640 10 monclient(hunting): picked mon.noname-b con 0x7f87b00b0770 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.862+0000 7f87b64fb640 10 monclient(hunting): picked mon.noname-a con 0x7f87b014a700 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87b64fb640 10 monclient(hunting): picked mon.noname-c con 0x7f87b00ac5a0 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87b64fb640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b00aba70 con 0x7f87b014a700 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87b64fb640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b00aecb0 con 0x7f87b00ac5a0 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87b64fb640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b00af150 con 0x7f87b00b0770 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87b64fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87b64fb640 10 monclient(hunting): authenticate will time out at 1320.171146s 2024-03-29T11:28:18.852 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87af7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f87b014a700 0x7f87b00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:51417/0 (socket says 172.21.15.138:51417) 2024-03-29T11:28:18.853 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.863+0000 7f87af7fe640 1 -- 172.21.15.138:0/2465872870 learned_addr learned my addr 172.21.15.138:0/2465872870 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.865+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250700980 0 0) 0x7f87b00aba70 con 0x7f87b014a700 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.865+0000 7f87aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.865+0000 7f87aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.865+0000 7f87aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.865+0000 7f87aeffd640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.866+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879c003520 con 0x7f87b014a700 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.866+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33647643 0 0) 0x7f879c003520 con 0x7f87b014a700 2024-03-29T11:28:18.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.866+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 >> v1:172.21.15.138:6790/0 conn(0x7f87b00b0770 legacy=0x7f87b0149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:18.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 >> v1:172.21.15.152:6789/0 conn(0x7f87b00ac5a0 legacy=0x7f87b0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 1 monclient: found mon.noname-a 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b00ac9d0 con 0x7f87b014a700 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f87a0001a10 con 0x7f87b014a700 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87b64fb640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:28:18.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87b64fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.867+0000 7f87aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87aeffd640 10 monclient: dump: 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:18.860 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87a0001ca0 con 0x7f87b014a700 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87b64fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87aeffd640 1 -- 172.21.15.138:0/2465872870 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f87a0002030 con 0x7f87b014a700 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87b64fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.868+0000 7f87b64fb640 4 set_mon_vals no callback set 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.869+0000 7f87b64fb640 10 monclient: shutdown 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.869+0000 7f87b64fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.869+0000 7f87b64fb640 1 -- 172.21.15.138:0/2465872870 >> v1:172.21.15.138:6789/0 conn(0x7f87b014a700 legacy=0x7f87b00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.869+0000 7f87aeffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.869+0000 7f87b64fb640 1 -- 172.21.15.138:0/2465872870 shutdown_connections 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.869+0000 7f87b64fb640 1 -- 172.21.15.138:0/2465872870 >> 172.21.15.138:0/2465872870 conn(0x7f87b006b660 msgr2=0x7f87b006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.870+0000 7f87b64fb640 1 -- 172.21.15.138:0/2465872870 shutdown_connections 2024-03-29T11:28:18.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.870+0000 7f87b64fb640 1 -- 172.21.15.138:0/2465872870 wait complete. 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.870+0000 7f87b64fb640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.870+0000 7f87b64fb640 10 monclient: monmap: 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.863 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.871+0000 7f87b64fb640 1 librados: starting msgr at 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.871+0000 7f87b64fb640 1 librados: starting objecter 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.871+0000 7f87b64fb640 1 Processor -- start 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.871+0000 7f87b64fb640 1 -- start start 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.871+0000 7f87b64fb640 1 librados: setting wanted keys 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.871+0000 7f87b64fb640 1 librados: calling monclient init 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.871+0000 7f87b64fb640 10 monclient: init 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.872+0000 7f87b64fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.872+0000 7f87b64fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:18.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.872+0000 7f87b64fb640 10 monclient(hunting): picked mon.noname-a con 0x7f87b00ac5a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:18.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.874+0000 7f87b64fb640 10 monclient(hunting): picked mon.noname-c con 0x7f87b00b0770 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:18.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.874+0000 7f87b64fb640 10 monclient(hunting): picked mon.noname-b con 0x7f87b014a700 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:18.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.874+0000 7f87b64fb640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b01db5f0 con 0x7f87b00ac5a0 2024-03-29T11:28:18.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.874+0000 7f87b64fb640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b012a2c0 con 0x7f87b014a700 2024-03-29T11:28:18.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.874+0000 7f87b64fb640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b012a5a0 con 0x7f87b00b0770 2024-03-29T11:28:18.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.874+0000 7f87b64fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.874+0000 7f87b64fb640 10 monclient(hunting): authenticate will time out at 1320.182872s 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.878+0000 7f87af7fe640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f87b00b0770 0x7f87b01dbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:37547/0 (socket says 172.21.15.138:37547) 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.878+0000 7f87af7fe640 1 -- 172.21.15.138:0/4020708007 learned_addr learned my addr 172.21.15.138:0/4020708007 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.878+0000 7f87acff9640 10 client.?.objecter ms_handle_connect 0x7f87b00b0770 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.878+0000 7f87acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.878+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114954348 0 0) 0x7f87b012a5a0 con 0x7f87b00b0770 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.878+0000 7f87acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.879+0000 7f87acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.879+0000 7f87acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.879+0000 7f87acff9640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.879+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8790004eb0 con 0x7f87b00b0770 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.879+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3001480576 0 0) 0x7f8790004eb0 con 0x7f87b00b0770 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.879+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b012a5a0 con 0x7f87b00b0770 2024-03-29T11:28:18.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.879+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f87a0001a10 con 0x7f87b00b0770 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.880+0000 7f87acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.880+0000 7f87acff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.880+0000 7f87acff9640 10 monclient(hunting): dump: 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:18.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.880+0000 7f87acff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:18.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.880+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3145705663 0 0) 0x7f87b012a5a0 con 0x7f87b00b0770 2024-03-29T11:28:18.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.880+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 >> v1:172.21.15.152:6789/0 conn(0x7f87b014a700 legacy=0x7f87b0147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.880+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 >> v1:172.21.15.138:6789/0 conn(0x7f87b00ac5a0 legacy=0x7f87b0128f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 1 monclient: found mon.c 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b01255e0 con 0x7f87b00b0770 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.873 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.883+0000 7f87b64fb640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 10 monclient: _renew_subs 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 1 -- 172.21.15.138:0/4020708007 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b01dc1b0 con 0x7f87b00b0770 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 10 monclient: _renew_subs 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 1 -- 172.21.15.138:0/4020708007 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b0147c60 con 0x7f87b00b0770 2024-03-29T11:28:18.874 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.884+0000 7f87b64fb640 1 librados: init done 2024-03-29T11:28:18.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.887+0000 7f87a67fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.876 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.887+0000 7f87a67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.887+0000 7f87a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:18.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.887+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8794005080 con 0x7f87b00b0770 2024-03-29T11:28:18.877 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.889+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87a0004e10 con 0x7f87b00b0770 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.889+0000 7f87acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.889+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f87a0005260 con 0x7f87b00b0770 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.889+0000 7f87acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.889+0000 7f87acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.889+0000 7f87acff9640 10 monclient: dump: 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:18.879 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:18.880 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:18.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.889+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f87a0005480 con 0x7f87b00b0770 2024-03-29T11:28:18.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.890+0000 7f87ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:18.880 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.890+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f87a00850f0 con 0x7f87b00b0770 2024-03-29T11:28:18.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.890+0000 7f87acff9640 10 client.4571.objecter ms_dispatch 0x7f87b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:18.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.890+0000 7f87acff9640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:18.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.890+0000 7f87acff9640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:18.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.890+0000 7f87acff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:28:18.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.890+0000 7f87affff640 10 monclient: get_auth_request con 0x7f879006dda0 auth_method 0 2024-03-29T11:28:18.881 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.892+0000 7f87acff9640 10 client.4571.objecter ms_handle_connect 0x7f879006dda0 2024-03-29T11:28:18.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.899+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87a0050a50 con 0x7f87b00b0770 2024-03-29T11:28:18.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.899+0000 7f87acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.888 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:18.899+0000 7f87acff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:19.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.219+0000 7f87a67fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.219+0000 7f87a67fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.219+0000 7f87a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:19.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.219+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8794005330 con 0x7f87b00b0770 2024-03-29T11:28:19.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.221+0000 7f87acff9640 1 -- 172.21.15.138:0/4020708007 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f87a0052980 con 0x7f87b00b0770 2024-03-29T11:28:19.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.221+0000 7f87acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.221+0000 7f87acff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:19.213 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:19.213 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:19.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 10 librados: watch_flush enter 2024-03-29T11:28:19.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 10 librados: watch_flush exit 2024-03-29T11:28:19.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:28:19.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:28:19.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 >> v1:172.21.15.152:6816/3052585572 conn(0x7f879006dda0 legacy=0x7f8790070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 10 monclient: shutdown 2024-03-29T11:28:19.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:19.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 >> v1:172.21.15.138:6790/0 conn(0x7f87b00b0770 legacy=0x7f87b01dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 shutdown_connections 2024-03-29T11:28:19.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.227+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 >> 172.21.15.138:0/4020708007 conn(0x7f87b006b660 msgr2=0x7f87b006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:19.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.228+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 shutdown_connections 2024-03-29T11:28:19.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.228+0000 7f87a67fc640 1 -- 172.21.15.138:0/4020708007 wait complete. 2024-03-29T11:28:19.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:19.228+0000 7f87a67fc640 1 librados: shutdown 2024-03-29T11:28:19.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:26.043 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:26.206 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.216+0000 7fdac1190640 10 monclient: get_monmap_and_config 2024-03-29T11:28:26.206 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.216+0000 7fdac1190640 10 monclient: build_initial_monmap 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.216+0000 7fdac1190640 10 monclient: monmap: 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:26.217820+0000 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:26.217820+0000 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:26.207 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:26.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.217+0000 7fdac1190640 1 Processor -- start 2024-03-29T11:28:26.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.217+0000 7fdac1190640 1 -- start start 2024-03-29T11:28:26.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.217+0000 7fdac1190640 10 monclient: init 2024-03-29T11:28:26.208 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.218+0000 7fdac1190640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:26.209 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.218+0000 7fdac1190640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:26.209 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.218+0000 7fdac1190640 10 monclient(hunting): picked mon.noname-b con 0x7fdabc0b1f60 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:26.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdac1190640 10 monclient(hunting): picked mon.noname-a con 0x7fdabc071eb0 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:26.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdac1190640 10 monclient(hunting): picked mon.noname-c con 0x7fdabc143d10 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:26.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdac1190640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc06ef80 con 0x7fdabc071eb0 2024-03-29T11:28:26.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdac1190640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc06f260 con 0x7fdabc143d10 2024-03-29T11:28:26.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdabb59e640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fdabc143d10 0x7fdabc146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:46256/0 (socket says 172.21.15.138:46256) 2024-03-29T11:28:26.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdabb59e640 1 -- 172.21.15.138:0/1761879861 learned_addr learned my addr 172.21.15.138:0/1761879861 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:26.210 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdac1190640 1 -- 172.21.15.138:0/1761879861 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc0b40c0 con 0x7fdabc0b1f60 2024-03-29T11:28:26.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdac1190640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:26.211 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.219+0000 7fdac1190640 10 monclient(hunting): authenticate will time out at 1327.529003s 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.220+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58599368 0 0) 0x7fdabc06f260 con 0x7fdabc143d10 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.221+0000 7fdab9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.221+0000 7fdab9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.221+0000 7fdab9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.221+0000 7fdab9d9b640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.221+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa4003520 con 0x7fdabc143d10 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.221+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005541284 0 0) 0x7fdaa4003520 con 0x7fdabc143d10 2024-03-29T11:28:26.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.221+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 >> v1:172.21.15.138:6790/0 conn(0x7fdabc0b1f60 legacy=0x7fdabc143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:26.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 >> v1:172.21.15.138:6789/0 conn(0x7fdabc071eb0 legacy=0x7fdabc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:28:26.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 1 monclient: found mon.noname-c 2024-03-29T11:28:26.216 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc153d50 con 0x7fdabc143d10 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdab4001a10 con 0x7fdabc143d10 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: dump: 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.217 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdac1190640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdac1190640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdab4001be0 con 0x7fdabc143d10 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 1 -- 172.21.15.138:0/1761879861 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdab4001ff0 con 0x7fdabc143d10 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdac1190640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdac1190640 4 set_mon_vals no callback set 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.223+0000 7fdab9d9b640 10 monclient: dump: 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.224+0000 7fdac1190640 10 monclient: shutdown 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.224+0000 7fdac1190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:26.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.224+0000 7fdac1190640 1 -- 172.21.15.138:0/1761879861 >> v1:172.21.15.152:6789/0 conn(0x7fdabc143d10 legacy=0x7fdabc146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.225+0000 7fdac1190640 1 -- 172.21.15.138:0/1761879861 shutdown_connections 2024-03-29T11:28:26.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.225+0000 7fdac1190640 1 -- 172.21.15.138:0/1761879861 >> 172.21.15.138:0/1761879861 conn(0x7fdabc066a00 msgr2=0x7fdabc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.227+0000 7fdac1190640 1 -- 172.21.15.138:0/1761879861 shutdown_connections 2024-03-29T11:28:26.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.228+0000 7fdac1190640 1 -- 172.21.15.138:0/1761879861 wait complete. 2024-03-29T11:28:26.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.229+0000 7fdac1190640 10 monclient: build_initial_monmap 2024-03-29T11:28:26.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.229+0000 7fdac1190640 10 monclient: monmap: 2024-03-29T11:28:26.221 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:26.221 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.230+0000 7fdac1190640 1 librados: starting msgr at 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.230+0000 7fdac1190640 1 librados: starting objecter 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.230+0000 7fdac1190640 1 Processor -- start 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.230+0000 7fdac1190640 1 -- start start 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.230+0000 7fdac1190640 1 librados: setting wanted keys 2024-03-29T11:28:26.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.230+0000 7fdac1190640 1 librados: calling monclient init 2024-03-29T11:28:26.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.230+0000 7fdac1190640 10 monclient: init 2024-03-29T11:28:26.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.231+0000 7fdac1190640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:26.223 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.231+0000 7fdac1190640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:26.225 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.231+0000 7fdac1190640 10 monclient(hunting): picked mon.noname-c con 0x7fdabc0b1f60 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:26.227 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdac1190640 10 monclient(hunting): picked mon.noname-a con 0x7fdabc143d10 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:26.227 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdac1190640 10 monclient(hunting): picked mon.noname-b con 0x7fdabc142040 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:26.227 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdac1190640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc071210 con 0x7fdabc143d10 2024-03-29T11:28:26.227 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdac1190640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc142440 con 0x7fdabc142040 2024-03-29T11:28:26.227 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdac1190640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc06f750 con 0x7fdabc0b1f60 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdac1190640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdac1190640 10 monclient(hunting): authenticate will time out at 1327.544945s 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdabad9d640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fdabc0b1f60 0x7fdabc138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:43489/0 (socket says 172.21.15.138:43489) 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.235+0000 7fdabad9d640 1 -- 172.21.15.138:0/3663103571 learned_addr learned my addr 172.21.15.138:0/3663103571 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 client.?.objecter ms_handle_connect 0x7fdabc0b1f60 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 client.?.objecter ms_handle_connect 0x7fdabc142040 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435913097 0 0) 0x7fdabc06f750 con 0x7fdabc0b1f60 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:28:26.228 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa0004eb0 con 0x7fdabc0b1f60 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405852731 0 0) 0x7fdabc142440 con 0x7fdabc142040 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa0006230 con 0x7fdabc142040 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990049365 0 0) 0x7fdaa0006230 con 0x7fdabc142040 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.236+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaa00066a0 con 0x7fdabc142040 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdab4001ef0 con 0x7fdabc142040 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 10 monclient(hunting): dump: 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:26.229 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1532718431 0 0) 0x7fdaa0004eb0 con 0x7fdabc0b1f60 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaa0006d20 con 0x7fdabc0b1f60 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 174696666 0 0) 0x7fdaa00066a0 con 0x7fdabc142040 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 >> v1:172.21.15.138:6790/0 conn(0x7fdabc0b1f60 legacy=0x7fdabc138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.230 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.237+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 >> v1:172.21.15.138:6789/0 conn(0x7fdabc143d10 legacy=0x7fdabc141930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:26.233 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:26.233 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 1 monclient: found mon.b 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc141230 con 0x7fdabc142040 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 10 monclient: _renew_subs 2024-03-29T11:28:26.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc141570 con 0x7fdabc142040 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 10 monclient: _renew_subs 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc135a70 con 0x7fdabc142040 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdac1190640 1 librados: init done 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdab40020d0 con 0x7fdabc142040 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.238+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdab4001ef0 con 0x7fdabc142040 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.239+0000 7fdaab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.239+0000 7fdaab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.239+0000 7fdaab7fe640 10 monclient: dump: 2024-03-29T11:28:26.235 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.239+0000 7fdab8d99640 4 set_mon_vals no callback set 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.241+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fdab401ac80 con 0x7fdabc142040 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fdab4084a30 con 0x7fdabc142040 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdaab7fe640 10 client.4462.objecter ms_dispatch 0x7fdabc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdaab7fe640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:26.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdaab7fe640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:26.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdaab7fe640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T11:28:26.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdac1190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdac1190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdac1190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:26.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.242+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda98005080 con 0x7fdabc142040 2024-03-29T11:28:26.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.243+0000 7fdabad9d640 10 monclient: get_auth_request con 0x7fdaa006e4c0 auth_method 0 2024-03-29T11:28:26.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.243+0000 7fdaab7fe640 10 client.4462.objecter ms_handle_connect 0x7fdaa006e4c0 2024-03-29T11:28:26.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.251+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdab4050310 con 0x7fdabc142040 2024-03-29T11:28:26.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.251+0000 7fdaab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.251+0000 7fdaab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:26.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.569+0000 7fdac1190640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.569+0000 7fdac1190640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.569+0000 7fdac1190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:28:26.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.569+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda98005330 con 0x7fdabc142040 2024-03-29T11:28:26.560 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.570+0000 7fdaab7fe640 1 -- 172.21.15.138:0/3663103571 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fdab4052240 con 0x7fdabc142040 2024-03-29T11:28:26.560 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.570+0000 7fdaab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.560 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.570+0000 7fdaab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:26.560 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:26.560 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:26.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 10 librados: watch_flush enter 2024-03-29T11:28:26.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 10 librados: watch_flush exit 2024-03-29T11:28:26.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T11:28:26.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 10 client.4462.objecter successfully canceled tick 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 >> v1:172.21.15.152:6816/3052585572 conn(0x7fdaa006e4c0 legacy=0x7fdaa0070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 10 monclient: shutdown 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.573+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 >> v1:172.21.15.152:6789/0 conn(0x7fdabc142040 legacy=0x7fdabc1e5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.574+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 shutdown_connections 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.574+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 >> 172.21.15.138:0/3663103571 conn(0x7fdabc066a00 msgr2=0x7fdabc146830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.574+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 shutdown_connections 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.574+0000 7fdac1190640 1 -- 172.21.15.138:0/3663103571 wait complete. 2024-03-29T11:28:26.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:26.574+0000 7fdac1190640 1 librados: shutdown 2024-03-29T11:28:26.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:33.387 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:33.551 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.561+0000 7ffacbe86640 10 monclient: get_monmap_and_config 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.561+0000 7ffacbe86640 10 monclient: build_initial_monmap 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.561+0000 7ffacbe86640 10 monclient: monmap: 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:33.562879+0000 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:33.562879+0000 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:33.552 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:33.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.563+0000 7ffacbe86640 1 Processor -- start 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.563+0000 7ffacbe86640 1 -- start start 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.563+0000 7ffacbe86640 10 monclient: init 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 10 monclient(hunting): picked mon.noname-c con 0x7ffac41457f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 10 monclient(hunting): picked mon.noname-a con 0x7ffac413f650 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 10 monclient(hunting): picked mon.noname-b con 0x7ffac40a9d70 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac40aaef0 con 0x7ffac413f650 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac4145bd0 con 0x7ffac41457f0 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac4145eb0 con 0x7ffac40a9d70 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.564+0000 7ffacbe86640 10 monclient(hunting): authenticate will time out at 1334.873401s 2024-03-29T11:28:33.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.566+0000 7ffac9c22640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ffac41457f0 0x7ffac413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:35590/0 (socket says 172.21.15.138:35590) 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.566+0000 7ffac9c22640 1 -- 172.21.15.138:0/2055121281 learned_addr learned my addr 172.21.15.138:0/2055121281 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.566+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152127286 0 0) 0x7ffac4145bd0 con 0x7ffac41457f0 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.567+0000 7ffac8c20640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.567+0000 7ffac8c20640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.567+0000 7ffac8c20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.567+0000 7ffac8c20640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.567+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab0003520 con 0x7ffac41457f0 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.571+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1467371736 0 0) 0x7ffab0003520 con 0x7ffac41457f0 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.571+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 >> v1:172.21.15.138:6790/0 conn(0x7ffac40a9d70 legacy=0x7ffac414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:33.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.571+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 >> v1:172.21.15.138:6789/0 conn(0x7ffac413f650 legacy=0x7ffac40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 10 monclient: _finish_hunting 0 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 1 monclient: found mon.noname-c 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4146340 con 0x7ffac41457f0 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 10 monclient: _finish_auth 0 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 10 monclient: _check_auth_tickets 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffabc001a10 con 0x7ffac41457f0 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:33.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.573+0000 7ffac8c20640 10 monclient: dump: 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.574+0000 7ffacbe86640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:28:33.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.574+0000 7ffacbe86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffabc001be0 con 0x7ffac41457f0 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffac8c20640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffac8c20640 1 -- 172.21.15.138:0/2055121281 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffabc001ff0 con 0x7ffac41457f0 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffacbe86640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffacbe86640 4 set_mon_vals no callback set 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffac8c20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffac8c20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.575+0000 7ffac8c20640 10 monclient: dump: 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:33.567 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.576+0000 7ffacbe86640 10 monclient: shutdown 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.576+0000 7ffacbe86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.576+0000 7ffacbe86640 1 -- 172.21.15.138:0/2055121281 >> v1:172.21.15.152:6789/0 conn(0x7ffac41457f0 legacy=0x7ffac413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.577+0000 7ffacbe86640 1 -- 172.21.15.138:0/2055121281 shutdown_connections 2024-03-29T11:28:33.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.577+0000 7ffacbe86640 1 -- 172.21.15.138:0/2055121281 >> 172.21.15.138:0/2055121281 conn(0x7ffac4066a00 msgr2=0x7ffac4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.578+0000 7ffacbe86640 1 -- 172.21.15.138:0/2055121281 shutdown_connections 2024-03-29T11:28:33.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.578+0000 7ffacbe86640 1 -- 172.21.15.138:0/2055121281 wait complete. 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.579+0000 7ffacbe86640 10 monclient: build_initial_monmap 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.579+0000 7ffacbe86640 10 monclient: monmap: 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:33.572 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:33.573 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:33.573 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:33.573 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.579+0000 7ffacbe86640 1 librados: starting msgr at 2024-03-29T11:28:33.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.579+0000 7ffacbe86640 1 librados: starting objecter 2024-03-29T11:28:33.574 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.580+0000 7ffacbe86640 1 Processor -- start 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.580+0000 7ffacbe86640 1 -- start start 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.580+0000 7ffacbe86640 1 librados: setting wanted keys 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.580+0000 7ffacbe86640 1 librados: calling monclient init 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.580+0000 7ffacbe86640 10 monclient: init 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 10 monclient(hunting): picked mon.noname-a con 0x7ffac40a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 10 monclient(hunting): picked mon.noname-b con 0x7ffac413f650 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 10 monclient(hunting): picked mon.noname-c con 0x7ffac41457f0 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac4142e10 con 0x7ffac40a9d70 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac4139ef0 con 0x7ffac413f650 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac413a1d0 con 0x7ffac41457f0 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:33.575 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.581+0000 7ffacbe86640 10 monclient(hunting): authenticate will time out at 1334.890329s 2024-03-29T11:28:33.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.585+0000 7ffac9421640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ffac413f650 0x7ffac413fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:37994/0 (socket says 172.21.15.138:37994) 2024-03-29T11:28:33.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.588+0000 7ffaca423640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7ffac41457f0 0x7ffac41eebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:36433/0 (socket says 172.21.15.138:36433) 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.588+0000 7ffaca423640 1 -- 172.21.15.138:0/2986370579 learned_addr learned my addr 172.21.15.138:0/2986370579 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.589+0000 7ffaba7fc640 10 client.?.objecter ms_handle_connect 0x7ffac41457f0 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.589+0000 7ffaba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.589+0000 7ffaba7fc640 10 client.?.objecter ms_handle_connect 0x7ffac40a9d70 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.589+0000 7ffaba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136164517 0 0) 0x7ffac4142e10 con 0x7ffac40a9d70 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8004eb0 con 0x7ffac40a9d70 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 client.?.objecter ms_handle_connect 0x7ffac413f650 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883279551 0 0) 0x7ffaa8004eb0 con 0x7ffac40a9d70 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaa8005300 con 0x7ffac40a9d70 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffabc001ef0 con 0x7ffac40a9d70 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): dump: 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:33.584 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802933925 0 0) 0x7ffac413a1d0 con 0x7ffac41457f0 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.585 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8004eb0 con 0x7ffac41457f0 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184490614 0 0) 0x7ffac4139ef0 con 0x7ffac413f650 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.590+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac413a1d0 con 0x7ffac413f650 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1817464577 0 0) 0x7ffaa8004eb0 con 0x7ffac41457f0 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffac4139ef0 con 0x7ffac41457f0 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffac0001820 con 0x7ffac41457f0 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:33.586 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 10 monclient(hunting): dump: 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:33.587 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4046280653 0 0) 0x7ffaa8005300 con 0x7ffac40a9d70 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 >> v1:172.21.15.138:6790/0 conn(0x7ffac41457f0 legacy=0x7ffac41eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 >> v1:172.21.15.152:6789/0 conn(0x7ffac413f650 legacy=0x7ffac413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 monclient: found mon.a 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:33.588 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4142910 con 0x7ffac40a9d70 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.591+0000 7ffaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 10 monclient: _renew_subs 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac41414c0 con 0x7ffac40a9d70 2024-03-29T11:28:33.589 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:33.590 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 10 monclient: _renew_subs 2024-03-29T11:28:33.590 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:33.590 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac4138a90 con 0x7ffac40a9d70 2024-03-29T11:28:33.590 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.592+0000 7ffacbe86640 1 librados: init done 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.594+0000 7ffacbe86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.594+0000 7ffacbe86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.594+0000 7ffacbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.594+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac4075f10 con 0x7ffac40a9d70 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.594+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffabc004e10 con 0x7ffac40a9d70 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.594+0000 7ffaba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.595+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ffabc0015d0 con 0x7ffac40a9d70 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.595+0000 7ffaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.595+0000 7ffaba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.595+0000 7ffaba7fc640 10 monclient: dump: 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:33.592 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.595+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ffabc01af90 con 0x7ffac40a9d70 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.595+0000 7ffabb7fe640 4 set_mon_vals no callback set 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.596+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7ffabc084db0 con 0x7ffac40a9d70 2024-03-29T11:28:33.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.597+0000 7ffaba7fc640 10 client.4506.objecter ms_dispatch 0x7ffac4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:33.594 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.597+0000 7ffaba7fc640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:33.594 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.597+0000 7ffaba7fc640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:33.594 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.597+0000 7ffac9421640 10 monclient: get_auth_request con 0x7ffaa806e180 auth_method 0 2024-03-29T11:28:33.594 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.597+0000 7ffaba7fc640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T11:28:33.594 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.597+0000 7ffaba7fc640 10 client.4506.objecter ms_handle_connect 0x7ffaa806e180 2024-03-29T11:28:33.595 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.604+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffabc0507c0 con 0x7ffac40a9d70 2024-03-29T11:28:33.595 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.604+0000 7ffaba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.595 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.604+0000 7ffaba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:33.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.965+0000 7ffacbe86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.965+0000 7ffacbe86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.965+0000 7ffacbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:33.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.965+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffac4138fe0 con 0x7ffac40a9d70 2024-03-29T11:28:33.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.970+0000 7ffaba7fc640 1 -- 172.21.15.138:0/2986370579 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7ffabc052670 con 0x7ffac40a9d70 2024-03-29T11:28:33.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.970+0000 7ffaba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.961 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.970+0000 7ffaba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:33.962 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:33.962 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:33.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.974+0000 7ffacbe86640 10 librados: watch_flush enter 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 10 librados: watch_flush exit 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 10 client.4506.objecter successfully canceled tick 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 >> v1:172.21.15.152:6816/3052585572 conn(0x7ffaa806e180 legacy=0x7ffaa8070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 10 monclient: shutdown 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 >> v1:172.21.15.138:6789/0 conn(0x7ffac40a9d70 legacy=0x7ffac413a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 shutdown_connections 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.975+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 >> 172.21.15.138:0/2986370579 conn(0x7ffac4066a00 msgr2=0x7ffac41d2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.976+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 shutdown_connections 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.976+0000 7ffacbe86640 1 -- 172.21.15.138:0/2986370579 wait complete. 2024-03-29T11:28:33.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:33.976+0000 7ffacbe86640 1 librados: shutdown 2024-03-29T11:28:33.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:40.796 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:40.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.953+0000 7f9f4a1a8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.953+0000 7f9f4a1a8640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.953+0000 7f9f4a1a8640 10 monclient: monmap: 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:40.954501+0000 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:40.954501+0000 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:40.944 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:40.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.954+0000 7f9f4a1a8640 1 Processor -- start 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.954+0000 7f9f4a1a8640 1 -- start start 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.954+0000 7f9f4a1a8640 10 monclient: init 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.955+0000 7f9f4a1a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.955+0000 7f9f4a1a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.955+0000 7f9f4a1a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f441457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.955+0000 7f9f4a1a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f4413f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.955+0000 7f9f437fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f9f441457f0 0x7f9f4413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:41454/0 (socket says 172.21.15.138:41454) 2024-03-29T11:28:40.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f42ffd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f9f4413f650 0x7f9f440b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:54364/0 (socket says 172.21.15.138:54364) 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f437fe640 1 -- 172.21.15.138:0/410880178 learned_addr learned my addr 172.21.15.138:0/410880178 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f4a1a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f440a9d70 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f440aaef0 con 0x7f9f441457f0 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f44145bd0 con 0x7f9f440a9d70 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f44145eb0 con 0x7f9f4413f650 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f4a1a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f4a1a8640 10 monclient(hunting): authenticate will time out at 1342.265741s 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738887493 0 0) 0x7f9f44145bd0 con 0x7f9f440a9d70 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f2c003520 con 0x7f9f440a9d70 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854836623 0 0) 0x7f9f440aaef0 con 0x7f9f441457f0 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.956+0000 7f9f427fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f44145bd0 con 0x7f9f441457f0 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072028929 0 0) 0x7f9f44145bd0 con 0x7f9f441457f0 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 >> v1:172.21.15.138:6790/0 conn(0x7f9f4413f650 legacy=0x7f9f440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 >> v1:172.21.15.152:6789/0 conn(0x7f9f440a9d70 legacy=0x7f9f4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 monclient: found mon.noname-a 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f44146340 con 0x7f9f441457f0 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9f34001670 con 0x7f9f441457f0 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: dump: 2024-03-29T11:28:40.951 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f4a1a8640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f4a1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:40.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f340017d0 con 0x7f9f441457f0 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 1 -- 172.21.15.138:0/410880178 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9f34001c20 con 0x7f9f441457f0 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f427fc640 10 monclient: dump: 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:40.953 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:40.954 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:40.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f4a1a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:40.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.957+0000 7f9f4a1a8640 4 set_mon_vals no callback set 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.958+0000 7f9f4a1a8640 10 monclient: shutdown 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.958+0000 7f9f4a1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.958+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 >> v1:172.21.15.138:6789/0 conn(0x7f9f441457f0 legacy=0x7f9f4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.958+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 shutdown_connections 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.958+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 >> 172.21.15.138:0/410880178 conn(0x7f9f44066a00 msgr2=0x7f9f44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.959+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 shutdown_connections 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.959+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/410880178 wait complete. 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 10 monclient: monmap: 2024-03-29T11:28:40.962 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 1 librados: starting msgr at 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 1 librados: starting objecter 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 1 Processor -- start 2024-03-29T11:28:40.963 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 1 -- start start 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 1 librados: setting wanted keys 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 1 librados: calling monclient init 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.960+0000 7f9f4a1a8640 10 monclient: init 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.961+0000 7f9f4a1a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.961+0000 7f9f4a1a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f440a9d70 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f4413f650 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f42ffd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f9f4413f650 0x7f9f4406f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:54646/0 (socket says 172.21.15.138:54646) 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f42ffd640 1 -- 172.21.15.138:0/1653631045 learned_addr learned my addr 172.21.15.138:0/1653631045 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f2bfff640 10 client.?.objecter ms_handle_connect 0x7f9f4413f650 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f441457f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4406ecf0 con 0x7f9f441457f0 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4406ea10 con 0x7f9f440a9d70 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f44070840 con 0x7f9f4413f650 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.962+0000 7f9f4a1a8640 10 monclient(hunting): authenticate will time out at 1342.271642s 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 client.?.objecter ms_handle_connect 0x7f9f441457f0 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292100639 0 0) 0x7f9f44070840 con 0x7f9f4413f650 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f30004eb0 con 0x7f9f4413f650 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 client.?.objecter ms_handle_connect 0x7f9f440a9d70 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430822664 0 0) 0x7f9f4406ecf0 con 0x7f9f441457f0 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f44070840 con 0x7f9f441457f0 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2364895011 0 0) 0x7f9f30004eb0 con 0x7f9f4413f650 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f4406ecf0 con 0x7f9f4413f650 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9f38001950 con 0x7f9f4413f650 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): dump: 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.963+0000 7f9f2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482236931 0 0) 0x7f9f4406ea10 con 0x7f9f440a9d70 2024-03-29T11:28:40.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f30004eb0 con 0x7f9f440a9d70 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2129183543 0 0) 0x7f9f4406ecf0 con 0x7f9f4413f650 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 >> v1:172.21.15.152:6789/0 conn(0x7f9f440a9d70 legacy=0x7f9f44071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 >> v1:172.21.15.138:6789/0 conn(0x7f9f441457f0 legacy=0x7f9f4406f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 monclient: found mon.c 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f44070d90 con 0x7f9f4413f650 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 10 monclient: _renew_subs 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f44071f10 con 0x7f9f4413f650 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 10 monclient: _renew_subs 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f441e7210 con 0x7f9f4413f650 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f4a1a8640 1 librados: init done 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f38001b30 con 0x7f9f4413f650 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9f38004c30 con 0x7f9f4413f650 2024-03-29T11:28:40.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f2bfff640 10 monclient: dump: 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.964+0000 7f9f41ffb640 4 set_mon_vals no callback set 2024-03-29T11:28:40.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.966+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f9f3801ac00 con 0x7f9f4413f650 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f4a1a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f4a1a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f4a1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f24005080 con 0x7f9f4413f650 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f9f38084af0 con 0x7f9f4413f650 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f2bfff640 10 client.4583.objecter ms_dispatch 0x7f9f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f2bfff640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f2bfff640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f2bfff640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.967+0000 7f9f437fe640 10 monclient: get_auth_request con 0x7f9f3006de20 auth_method 0 2024-03-29T11:28:40.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.970+0000 7f9f2bfff640 10 client.4583.objecter ms_handle_connect 0x7f9f3006de20 2024-03-29T11:28:40.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.976+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f38050430 con 0x7f9f4413f650 2024-03-29T11:28:40.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.976+0000 7f9f2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:40.976+0000 7f9f2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:41.365 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.374+0000 7f9f4a1a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.365 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.374+0000 7f9f4a1a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.365 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.374+0000 7f9f4a1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:28:41.365 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.374+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f24005330 con 0x7f9f4413f650 2024-03-29T11:28:41.366 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.375+0000 7f9f2bfff640 1 -- 172.21.15.138:0/1653631045 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f9f380841a0 con 0x7f9f4413f650 2024-03-29T11:28:41.366 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.375+0000 7f9f2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.366 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.375+0000 7f9f2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:41.366 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:41.366 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.379+0000 7f9f4a1a8640 10 librados: watch_flush enter 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.379+0000 7f9f4a1a8640 10 librados: watch_flush exit 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.379+0000 7f9f4a1a8640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.379+0000 7f9f4a1a8640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.379+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 >> v1:172.21.15.152:6816/3052585572 conn(0x7f9f3006de20 legacy=0x7f9f300702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 10 monclient: shutdown 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 >> v1:172.21.15.138:6790/0 conn(0x7f9f4413f650 legacy=0x7f9f4406f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.370 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 shutdown_connections 2024-03-29T11:28:41.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 >> 172.21.15.138:0/1653631045 conn(0x7f9f44066a00 msgr2=0x7f9f44066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:41.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 shutdown_connections 2024-03-29T11:28:41.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 1 -- 172.21.15.138:0/1653631045 wait complete. 2024-03-29T11:28:41.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:41.380+0000 7f9f4a1a8640 1 librados: shutdown 2024-03-29T11:28:41.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:48.194 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.348+0000 7fcf3837b640 10 monclient: get_monmap_and_config 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.348+0000 7fcf3837b640 10 monclient: build_initial_monmap 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.348+0000 7fcf3837b640 10 monclient: monmap: 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:48.349564+0000 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:48.349564+0000 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:48.339 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:48.340 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:48.340 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:48.340 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:48.340 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:48.340 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.349+0000 7fcf3837b640 1 Processor -- start 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.349+0000 7fcf3837b640 1 -- start start 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.349+0000 7fcf3837b640 10 monclient: init 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 10 monclient(hunting): picked mon.noname-b con 0x7fcf30071de0 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 10 monclient(hunting): picked mon.noname-c con 0x7fcf300721c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf30145ae0 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf30145650 con 0x7fcf30145ae0 2024-03-29T11:28:48.341 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300ae260 con 0x7fcf300721c0 2024-03-29T11:28:48.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf301420d0 con 0x7fcf30071de0 2024-03-29T11:28:48.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:48.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf3837b640 10 monclient(hunting): authenticate will time out at 1349.659794s 2024-03-29T11:28:48.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf35916640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fcf300721c0 0x7fcf3013f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:36236/0 (socket says 172.21.15.138:36236) 2024-03-29T11:28:48.342 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.350+0000 7fcf35916640 1 -- 172.21.15.138:0/1821638861 learned_addr learned my addr 172.21.15.138:0/1821638861 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.351+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842611401 0 0) 0x7fcf300ae260 con 0x7fcf300721c0 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.351+0000 7fcf35115640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.351+0000 7fcf35115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.351+0000 7fcf35115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.351+0000 7fcf35115640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.351+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c003520 con 0x7fcf300721c0 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24691360 0 0) 0x7fcf301420d0 con 0x7fcf30071de0 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.343 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf300ae260 con 0x7fcf30071de0 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516142539 0 0) 0x7fcf30145650 con 0x7fcf30145ae0 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c0039d0 con 0x7fcf30145ae0 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443700419 0 0) 0x7fcf300ae260 con 0x7fcf30071de0 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 >> v1:172.21.15.152:6789/0 conn(0x7fcf300721c0 legacy=0x7fcf3013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 >> v1:172.21.15.138:6789/0 conn(0x7fcf30145ae0 legacy=0x7fcf30145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: _finish_hunting 0 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 monclient: found mon.noname-b 2024-03-29T11:28:48.344 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300b0590 con 0x7fcf30071de0 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: _finish_auth 0 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: _check_auth_tickets 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf3837b640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fcf200015f0 con 0x7fcf30071de0 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf3837b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: dump: 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.345 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf3837b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf200017d0 con 0x7fcf30071de0 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf35115640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:48.346 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf3837b640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.352+0000 7fcf3837b640 4 set_mon_vals no callback set 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.353+0000 7fcf35115640 1 -- 172.21.15.138:0/1821638861 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fcf200015f0 con 0x7fcf30071de0 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.353+0000 7fcf35115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.353+0000 7fcf35115640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.353+0000 7fcf35115640 10 monclient: dump: 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:48.347 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:48.348 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:48.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.353+0000 7fcf3837b640 10 monclient: shutdown 2024-03-29T11:28:48.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.353+0000 7fcf3837b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:48.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.353+0000 7fcf3837b640 1 -- 172.21.15.138:0/1821638861 >> v1:172.21.15.138:6790/0 conn(0x7fcf30071de0 legacy=0x7fcf300aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.354+0000 7fcf3837b640 1 -- 172.21.15.138:0/1821638861 shutdown_connections 2024-03-29T11:28:48.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.354+0000 7fcf3837b640 1 -- 172.21.15.138:0/1821638861 >> 172.21.15.138:0/1821638861 conn(0x7fcf3006b660 msgr2=0x7fcf3006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.354+0000 7fcf3837b640 1 -- 172.21.15.138:0/1821638861 shutdown_connections 2024-03-29T11:28:48.354 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.354+0000 7fcf3837b640 1 -- 172.21.15.138:0/1821638861 wait complete. 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.355+0000 7fcf3837b640 10 monclient: build_initial_monmap 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.355+0000 7fcf3837b640 10 monclient: monmap: 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:48.355 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.355+0000 7fcf3837b640 1 librados: starting msgr at 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.355+0000 7fcf3837b640 1 librados: starting objecter 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.355+0000 7fcf3837b640 1 Processor -- start 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 1 -- start start 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 1 librados: setting wanted keys 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 1 librados: calling monclient init 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient: init 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient(hunting): picked mon.noname-c con 0x7fcf30071de0 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient(hunting): picked mon.noname-b con 0x7fcf30145ae0 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3006f760 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300ad480 con 0x7fcf3006f760 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3006dd90 con 0x7fcf30145ae0 2024-03-29T11:28:48.356 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3006ed80 con 0x7fcf30071de0 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.356+0000 7fcf3837b640 10 monclient(hunting): authenticate will time out at 1349.665997s 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf36918640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fcf3006f760 0x7fcf301ef540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:60726/0 (socket says 172.21.15.138:60726) 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf36918640 1 -- 172.21.15.138:0/3674069092 learned_addr learned my addr 172.21.15.138:0/3674069092 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 client.?.objecter ms_handle_connect 0x7fcf3006f760 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 client.?.objecter ms_handle_connect 0x7fcf30071de0 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694634492 0 0) 0x7fcf300ad480 con 0x7fcf3006f760 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.357+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf14004eb0 con 0x7fcf3006f760 2024-03-29T11:28:48.357 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.358+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3235998424 0 0) 0x7fcf14004eb0 con 0x7fcf3006f760 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.358+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf300ad480 con 0x7fcf3006f760 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.358+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fcf2c001c90 con 0x7fcf3006f760 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.358+0000 7fcf26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.358+0000 7fcf26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.358+0000 7fcf26ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:48.358 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.358+0000 7fcf26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1719560157 0 0) 0x7fcf300ad480 con 0x7fcf3006f760 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 >> v1:172.21.15.138:6790/0 conn(0x7fcf30071de0 legacy=0x7fcf301349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 >> v1:172.21.15.152:6789/0 conn(0x7fcf30145ae0 legacy=0x7fcf301ebe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 1 monclient: found mon.a 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3006f060 con 0x7fcf3006f760 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:48.359 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 10 monclient: _renew_subs 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3006f340 con 0x7fcf3006f760 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 10 monclient: _renew_subs 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3006fb70 con 0x7fcf3006f760 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.359+0000 7fcf3837b640 1 librados: init done 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.360+0000 7fcf26ffd640 10 client.4518.objecter ms_handle_connect 0x7fcf30145ae0 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.360+0000 7fcf26ffd640 10 client.4518.objecter resend_mon_ops 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.360+0000 7fcf3837b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.360+0000 7fcf3837b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.360 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.360+0000 7fcf3837b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.360+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf30075f10 con 0x7fcf3006f760 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.361+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf2c001eb0 con 0x7fcf3006f760 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.361+0000 7fcf26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.361+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fcf2c003de0 con 0x7fcf3006f760 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.361+0000 7fcf26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.361+0000 7fcf34914640 4 set_mon_vals no callback set 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.362+0000 7fcf26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.362+0000 7fcf26ffd640 10 monclient: dump: 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:48.361 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.362 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:48.362 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:48.362 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:48.362 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:48.362 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:48.362 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.363+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fcf2c003fc0 con 0x7fcf3006f760 2024-03-29T11:28:48.363 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.364+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fcf2c084ef0 con 0x7fcf3006f760 2024-03-29T11:28:48.363 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.364+0000 7fcf36117640 10 monclient: get_auth_request con 0x7fcf1406dbb0 auth_method 0 2024-03-29T11:28:48.363 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.364+0000 7fcf26ffd640 10 client.4518.objecter ms_dispatch 0x7fcf30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:48.363 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.364+0000 7fcf26ffd640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:48.364 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.364+0000 7fcf26ffd640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:48.364 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.364+0000 7fcf26ffd640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:28:48.364 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.366+0000 7fcf26ffd640 10 client.4518.objecter ms_handle_connect 0x7fcf1406dbb0 2024-03-29T11:28:48.364 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.370+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf2c0507d0 con 0x7fcf3006f760 2024-03-29T11:28:48.364 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.370+0000 7fcf26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.364 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.370+0000 7fcf26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:48.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.724+0000 7fcf3837b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.724+0000 7fcf3837b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.724+0000 7fcf3837b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:48.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.724+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf3006ab70 con 0x7fcf3006f760 2024-03-29T11:28:48.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.725+0000 7fcf26ffd640 1 -- 172.21.15.138:0/3674069092 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fcf2c052700 con 0x7fcf3006f760 2024-03-29T11:28:48.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.725+0000 7fcf26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.725+0000 7fcf26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:48.716 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:48.716 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 10 librados: watch_flush enter 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 10 librados: watch_flush exit 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 >> v1:172.21.15.152:6816/3052585572 conn(0x7fcf1406dbb0 legacy=0x7fcf14070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 10 monclient: shutdown 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 >> v1:172.21.15.138:6789/0 conn(0x7fcf3006f760 legacy=0x7fcf301ef540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 shutdown_connections 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 >> 172.21.15.138:0/3674069092 conn(0x7fcf3006b660 msgr2=0x7fcf301386f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.729+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 shutdown_connections 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.730+0000 7fcf3837b640 1 -- 172.21.15.138:0/3674069092 wait complete. 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:48.730+0000 7fcf3837b640 1 librados: shutdown 2024-03-29T11:28:48.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:55.543 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.704+0000 7f43767b6640 10 monclient: get_monmap_and_config 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.704+0000 7f43767b6640 10 monclient: build_initial_monmap 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.704+0000 7f43767b6640 10 monclient: monmap: 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:28:55.705360+0000 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:28:55.705360+0000 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:28:55.695 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:28:55.696 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:55.697 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.706+0000 7f43767b6640 1 Processor -- start 2024-03-29T11:28:55.697 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.706+0000 7f43767b6640 1 -- start start 2024-03-29T11:28:55.697 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.706+0000 7f43767b6640 10 monclient: init 2024-03-29T11:28:55.697 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 10 monclient(hunting): picked mon.noname-a con 0x7f43700b1f60 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 10 monclient(hunting): picked mon.noname-c con 0x7f4370071eb0 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 10 monclient(hunting): picked mon.noname-b con 0x7f4370143d10 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437006ef80 con 0x7f43700b1f60 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437006f260 con 0x7f4370071eb0 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700b40c0 con 0x7f4370143d10 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:55.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.707+0000 7f43767b6640 10 monclient(hunting): authenticate will time out at 1357.016418s 2024-03-29T11:28:55.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.710+0000 7f436ffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f43700b1f60 0x7f4370143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:51269/0 (socket says 172.21.15.138:51269) 2024-03-29T11:28:55.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.710+0000 7f436ffff640 1 -- 172.21.15.138:0/2754518955 learned_addr learned my addr 172.21.15.138:0/2754518955 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:55.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140438739 0 0) 0x7f437006ef80 con 0x7f43700b1f60 2024-03-29T11:28:55.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:28:55.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4358003520 con 0x7f43700b1f60 2024-03-29T11:28:55.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2577630744 0 0) 0x7f4358003520 con 0x7f43700b1f60 2024-03-29T11:28:55.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 >> v1:172.21.15.138:6790/0 conn(0x7f4370143d10 legacy=0x7f4370146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:55.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 >> v1:172.21.15.152:6789/0 conn(0x7f4370071eb0 legacy=0x7f4370146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:55.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:55.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 monclient: found mon.noname-a 2024-03-29T11:28:55.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:55.703 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370155d70 con 0x7f43700b1f60 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient: _finish_auth 0 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4368001a10 con 0x7f43700b1f60 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.711+0000 7f436effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 10 monclient: dump: 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:55.704 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4368001c70 con 0x7f43700b1f60 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 1 -- 172.21.15.138:0/2754518955 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4368001a10 con 0x7f43700b1f60 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.712+0000 7f436effd640 10 monclient: dump: 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.705 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.714+0000 7f43767b6640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.714+0000 7f43767b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.714+0000 7f43767b6640 4 set_mon_vals no callback set 2024-03-29T11:28:55.706 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.716+0000 7f43767b6640 10 monclient: shutdown 2024-03-29T11:28:55.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.716+0000 7f43767b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:55.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.716+0000 7f43767b6640 1 -- 172.21.15.138:0/2754518955 >> v1:172.21.15.138:6789/0 conn(0x7f43700b1f60 legacy=0x7f4370143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.708 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.718+0000 7f43767b6640 1 -- 172.21.15.138:0/2754518955 shutdown_connections 2024-03-29T11:28:55.708 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.718+0000 7f43767b6640 1 -- 172.21.15.138:0/2754518955 >> 172.21.15.138:0/2754518955 conn(0x7f4370066a00 msgr2=0x7f4370066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.709 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.718+0000 7f43767b6640 1 -- 172.21.15.138:0/2754518955 shutdown_connections 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.718+0000 7f43767b6640 1 -- 172.21.15.138:0/2754518955 wait complete. 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.719+0000 7f43767b6640 10 monclient: build_initial_monmap 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.719+0000 7f43767b6640 10 monclient: monmap: 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:28:55.711 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.719+0000 7f43767b6640 1 librados: starting msgr at 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.719+0000 7f43767b6640 1 librados: starting objecter 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.719+0000 7f43767b6640 1 Processor -- start 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.720+0000 7f43767b6640 1 -- start start 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.720+0000 7f43767b6640 1 librados: setting wanted keys 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.720+0000 7f43767b6640 1 librados: calling monclient init 2024-03-29T11:28:55.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.720+0000 7f43767b6640 10 monclient: init 2024-03-29T11:28:55.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.721+0000 7f43767b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:55.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.721+0000 7f43767b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:55.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.721+0000 7f43767b6640 10 monclient(hunting): picked mon.noname-b con 0x7f43700b1f60 addr v1:172.21.15.152:6789/0 2024-03-29T11:28:55.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.721+0000 7f43767b6640 10 monclient(hunting): picked mon.noname-c con 0x7f4370143d10 addr v1:172.21.15.138:6790/0 2024-03-29T11:28:55.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.724+0000 7f43767b6640 10 monclient(hunting): picked mon.noname-a con 0x7f4370154c80 addr v1:172.21.15.138:6789/0 2024-03-29T11:28:55.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.724+0000 7f43767b6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700aa7e0 con 0x7f4370154c80 2024-03-29T11:28:55.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.724+0000 7f43767b6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700ab8e0 con 0x7f43700b1f60 2024-03-29T11:28:55.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.724+0000 7f43767b6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700ac690 con 0x7f4370143d10 2024-03-29T11:28:55.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.724+0000 7f43767b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:55.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.724+0000 7f43767b6640 10 monclient(hunting): authenticate will time out at 1357.033606s 2024-03-29T11:28:55.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.727+0000 7f436ffff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f43700b1f60 0x7f43700aa0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:58269/0 (socket says 172.21.15.138:58269) 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.727+0000 7f436ffff640 1 -- 172.21.15.138:0/2564880721 learned_addr learned my addr 172.21.15.138:0/2564880721 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.727+0000 7f436cff9640 10 client.?.objecter ms_handle_connect 0x7f4370154c80 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.727+0000 7f436cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.727+0000 7f436cff9640 10 client.?.objecter ms_handle_connect 0x7f43700b1f60 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.727+0000 7f436cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977260474 0 0) 0x7f43700aa7e0 con 0x7f4370154c80 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c004eb0 con 0x7f4370154c80 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442715458 0 0) 0x7f43700ab8e0 con 0x7f43700b1f60 2024-03-29T11:28:55.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43700aa7e0 con 0x7f43700b1f60 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3761792998 0 0) 0x7f435c004eb0 con 0x7f4370154c80 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.728+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43700ab8e0 con 0x7f4370154c80 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4360001750 con 0x7f4370154c80 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient(hunting): dump: 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.723 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554505675 0 0) 0x7f43700aa7e0 con 0x7f43700b1f60 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f435c004eb0 con 0x7f43700b1f60 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 client.?.objecter ms_handle_connect 0x7f4370143d10 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4368002040 con 0x7f43700b1f60 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:55.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient(hunting): dump: 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2768391424 0 0) 0x7f43700ab8e0 con 0x7f4370154c80 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 >> v1:172.21.15.138:6790/0 conn(0x7f4370143d10 legacy=0x7f43700ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 >> v1:172.21.15.152:6789/0 conn(0x7f43700b1f60 legacy=0x7f43700aa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 monclient: found mon.a 2024-03-29T11:28:55.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:55.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:55.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.729+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700aacb0 con 0x7f4370154c80 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f436cff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f436cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f436cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 10 monclient: _renew_subs 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43701404d0 con 0x7f4370154c80 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 10 monclient: _renew_subs 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:55.729 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43701e7130 con 0x7f4370154c80 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.730+0000 7f43767b6640 1 librados: init done 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.732+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43600018b0 con 0x7f4370154c80 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.732+0000 7f436cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.732+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f4360003790 con 0x7f4370154c80 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.732+0000 7f436cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.732+0000 7f436cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.732+0000 7f436cff9640 10 monclient: dump: 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.730 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.732+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f43600039b0 con 0x7f4370154c80 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.733+0000 7f436e7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.733+0000 7f436ffff640 10 monclient: get_auth_request con 0x7f435c06e0a0 auth_method 0 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.733+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f4360083700 con 0x7f4370154c80 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.733+0000 7f436cff9640 10 client.4524.objecter ms_dispatch 0x7f4370000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.733+0000 7f436cff9640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.733+0000 7f436cff9640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.733+0000 7f436cff9640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.734+0000 7f436cff9640 10 client.4524.objecter ms_handle_connect 0x7f435c06e0a0 2024-03-29T11:28:55.731 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.734+0000 7f43567fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.734+0000 7f43567fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.734+0000 7f43567fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:55.732 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.734+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4350005080 con 0x7f4370154c80 2024-03-29T11:28:55.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.745+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f436004f060 con 0x7f4370154c80 2024-03-29T11:28:55.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.745+0000 7f436cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:55.745+0000 7f436cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:56.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.069+0000 7f43567fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.069+0000 7f43567fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.069+0000 7f43567fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:28:56.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.069+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4350005330 con 0x7f4370154c80 2024-03-29T11:28:56.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.072+0000 7f436cff9640 1 -- 172.21.15.138:0/2564880721 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f4360050f90 con 0x7f4370154c80 2024-03-29T11:28:56.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.072+0000 7f436cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.072+0000 7f436cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:56.064 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:28:56.064 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 10 librados: watch_flush enter 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 10 librados: watch_flush exit 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 10 client.4524.objecter successfully canceled tick 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 >> v1:172.21.15.152:6816/3052585572 conn(0x7f435c06e0a0 legacy=0x7f435c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 10 monclient: shutdown 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.076+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 >> v1:172.21.15.138:6789/0 conn(0x7f4370154c80 legacy=0x7f437013fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.077+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 shutdown_connections 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.077+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 >> 172.21.15.138:0/2564880721 conn(0x7f4370066a00 msgr2=0x7f4370066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.077+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 shutdown_connections 2024-03-29T11:28:56.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.077+0000 7f43567fc640 1 -- 172.21.15.138:0/2564880721 wait complete. 2024-03-29T11:28:56.072 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:28:56.079+0000 7f43567fc640 1 librados: shutdown 2024-03-29T11:28:56.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:02.899 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:03.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.054+0000 7f86a1f2b640 10 monclient: get_monmap_and_config 2024-03-29T11:29:03.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.054+0000 7f86a1f2b640 10 monclient: build_initial_monmap 2024-03-29T11:29:03.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.054+0000 7f86a1f2b640 10 monclient: monmap: 2024-03-29T11:29:03.044 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:03.054565+0000 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:03.054565+0000 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:03.045 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:03.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.055+0000 7f86a1f2b640 1 Processor -- start 2024-03-29T11:29:03.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.055+0000 7f86a1f2b640 1 -- start start 2024-03-29T11:29:03.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.055+0000 7f86a1f2b640 10 monclient: init 2024-03-29T11:29:03.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.056+0000 7f86a1f2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:03.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.056+0000 7f86a1f2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:03.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.056+0000 7f86a0f29640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f869c0b2800 0x7f869c142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:38546/0 (socket says 172.21.15.138:38546) 2024-03-29T11:29:03.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.056+0000 7f86a0f29640 1 -- 172.21.15.138:0/1151598362 learned_addr learned my addr 172.21.15.138:0/1151598362 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:03.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.056+0000 7f86a1f2b640 10 monclient(hunting): picked mon.noname-b con 0x7f869c0b2800 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:03.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.056+0000 7f86a1f2b640 10 monclient(hunting): picked mon.noname-a con 0x7f869c0a9c00 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:03.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.059+0000 7f86a1f2b640 10 monclient(hunting): picked mon.noname-c con 0x7f869c0b38c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:03.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.059+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c141eb0 con 0x7f869c0a9c00 2024-03-29T11:29:03.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.059+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0b22d0 con 0x7f869c0b38c0 2024-03-29T11:29:03.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.059+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c142d30 con 0x7f869c0b2800 2024-03-29T11:29:03.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.059+0000 7f86a1f2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:03.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.059+0000 7f86a1f2b640 10 monclient(hunting): authenticate will time out at 1364.367236s 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357630877 0 0) 0x7f869c142d30 con 0x7f869c0b2800 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c003520 con 0x7f869c0b2800 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024771596 0 0) 0x7f869c0b22d0 con 0x7f869c0b38c0 2024-03-29T11:29:03.051 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f869c142d30 con 0x7f869c0b38c0 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301592069 0 0) 0x7f869c141eb0 con 0x7f869c0a9c00 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.060+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c0039d0 con 0x7f869c0a9c00 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727981673 0 0) 0x7f868c003520 con 0x7f869c0b2800 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 >> v1:172.21.15.152:6789/0 conn(0x7f869c0b38c0 legacy=0x7f869c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 >> v1:172.21.15.138:6789/0 conn(0x7f869c0a9c00 legacy=0x7f869c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0aaf90 con 0x7f869c0b2800 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8690001690 con 0x7f869c0b2800 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: dump: 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:03.053 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f86a1f2b640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f86a1f2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86900017f0 con 0x7f869c0b2800 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 1 -- 172.21.15.138:0/1151598362 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8690001b80 con 0x7f869c0b2800 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f869b7fe640 10 monclient: dump: 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f86a1f2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:03.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.061+0000 7f86a1f2b640 4 set_mon_vals no callback set 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.062+0000 7f86a1f2b640 10 monclient: shutdown 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.062+0000 7f86a1f2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.062+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 >> v1:172.21.15.138:6790/0 conn(0x7f869c0b2800 legacy=0x7f869c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.063+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 shutdown_connections 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.063+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 >> 172.21.15.138:0/1151598362 conn(0x7f869c066a00 msgr2=0x7f869c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.063+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 shutdown_connections 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.064+0000 7f86a1f2b640 1 -- 172.21.15.138:0/1151598362 wait complete. 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.066+0000 7f86a1f2b640 10 monclient: build_initial_monmap 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.066+0000 7f86a1f2b640 10 monclient: monmap: 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:03.057 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.066+0000 7f86a1f2b640 1 librados: starting msgr at 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.066+0000 7f86a1f2b640 1 librados: starting objecter 2024-03-29T11:29:03.058 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.067+0000 7f86a1f2b640 1 Processor -- start 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.067+0000 7f86a1f2b640 1 -- start start 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.068+0000 7f86a1f2b640 1 librados: setting wanted keys 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.068+0000 7f86a1f2b640 1 librados: calling monclient init 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.068+0000 7f86a1f2b640 10 monclient: init 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.068+0000 7f86a1f2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 10 monclient(hunting): picked mon.noname-b con 0x7f869c0a9c00 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 10 monclient(hunting): picked mon.noname-a con 0x7f869c0b2800 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 10 monclient(hunting): picked mon.noname-c con 0x7f869c0b38c0 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c138830 con 0x7f869c0b2800 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c06fce0 con 0x7f869c0a9c00 2024-03-29T11:29:03.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c1db900 con 0x7f869c0b38c0 2024-03-29T11:29:03.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:03.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.069+0000 7f86a1f2b640 10 monclient(hunting): authenticate will time out at 1364.377149s 2024-03-29T11:29:03.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.072+0000 7f869bfff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f869c0b2800 0x7f869c137a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:36207/0 (socket says 172.21.15.138:36207) 2024-03-29T11:29:03.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.072+0000 7f869bfff640 1 -- 172.21.15.138:0/4165282126 learned_addr learned my addr 172.21.15.138:0/4165282126 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:03.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.072+0000 7f86997fa640 10 client.?.objecter ms_handle_connect 0x7f869c0b2800 2024-03-29T11:29:03.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.072+0000 7f86997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 client.?.objecter ms_handle_connect 0x7f869c0b38c0 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57877523 0 0) 0x7f869c1db900 con 0x7f869c0b38c0 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:29:03.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8680004eb0 con 0x7f869c0b38c0 2024-03-29T11:29:03.066 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284002740 0 0) 0x7f869c138830 con 0x7f869c0b2800 2024-03-29T11:29:03.066 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.066 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.066 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.066 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:29:03.066 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f869c1db900 con 0x7f869c0b2800 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 477818534 0 0) 0x7f8680004eb0 con 0x7f869c0b38c0 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f869c138830 con 0x7f869c0b38c0 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8694001bd0 con 0x7f869c0b38c0 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.073+0000 7f86997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.074+0000 7f86997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.074+0000 7f86997fa640 10 monclient(hunting): dump: 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:03.067 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.074+0000 7f86997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.074+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2756669940 0 0) 0x7f869c138830 con 0x7f869c0b38c0 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.074+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 >> v1:172.21.15.152:6789/0 conn(0x7f869c0a9c00 legacy=0x7f869c1dbe20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 >> v1:172.21.15.138:6789/0 conn(0x7f869c0b2800 legacy=0x7f869c137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 1 monclient: found mon.c 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c1dc530 con 0x7f869c0b38c0 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:03.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 10 monclient: _renew_subs 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c06de90 con 0x7f869c0b38c0 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 10 monclient: _renew_subs 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c13c870 con 0x7f869c0b38c0 2024-03-29T11:29:03.069 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.075+0000 7f86a1f2b640 1 librados: init done 2024-03-29T11:29:03.070 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.078+0000 7f86a1f2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.070 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.078+0000 7f86a1f2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.078+0000 7f86a1f2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.078+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c1dba90 con 0x7f869c0b38c0 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.079+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8694001d30 con 0x7f869c0b38c0 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.079+0000 7f86997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.079+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8694003b10 con 0x7f869c0b38c0 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.079+0000 7f86997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.079+0000 7f86997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.079+0000 7f86997fa640 10 monclient: dump: 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:03.071 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.072 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:03.072 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:03.072 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:03.072 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:03.072 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:03.072 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.079+0000 7f869affd640 4 set_mon_vals no callback set 2024-03-29T11:29:03.073 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.081+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f8694019ae0 con 0x7f869c0b38c0 2024-03-29T11:29:03.073 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.081+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f8694083ab0 con 0x7f869c0b38c0 2024-03-29T11:29:03.073 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.081+0000 7f86997fa640 10 client.4595.objecter ms_dispatch 0x7f869c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:03.073 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.081+0000 7f86997fa640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:03.073 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.081+0000 7f86997fa640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:03.073 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.082+0000 7f86997fa640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:29:03.073 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.083+0000 7f86a0f29640 10 monclient: get_auth_request con 0x7f868006dad0 auth_method 0 2024-03-29T11:29:03.074 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.084+0000 7f86997fa640 10 client.4595.objecter ms_handle_connect 0x7f868006dad0 2024-03-29T11:29:03.078 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.088+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f869404f1c0 con 0x7f869c0b38c0 2024-03-29T11:29:03.078 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.088+0000 7f86997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.078 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.088+0000 7f86997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:03.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.414+0000 7f86a1f2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.414+0000 7f86a1f2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.414+0000 7f86a1f2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:03.405 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.414+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f869c0aa340 con 0x7f869c0b38c0 2024-03-29T11:29:03.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.416+0000 7f86997fa640 1 -- 172.21.15.138:0/4165282126 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f86940512c0 con 0x7f869c0b38c0 2024-03-29T11:29:03.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.416+0000 7f86997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.407 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.416+0000 7f86997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:03.407 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:03.407 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:03.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.421+0000 7f86a1f2b640 10 librados: watch_flush enter 2024-03-29T11:29:03.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.421+0000 7f86a1f2b640 10 librados: watch_flush exit 2024-03-29T11:29:03.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:29:03.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:29:03.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 >> v1:172.21.15.152:6816/3052585572 conn(0x7f868006dad0 legacy=0x7f868006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 10 monclient: shutdown 2024-03-29T11:29:03.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:03.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 >> v1:172.21.15.138:6790/0 conn(0x7f869c0b38c0 legacy=0x7f869c138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 shutdown_connections 2024-03-29T11:29:03.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 >> 172.21.15.138:0/4165282126 conn(0x7f869c066a00 msgr2=0x7f869c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:03.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 shutdown_connections 2024-03-29T11:29:03.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.422+0000 7f86a1f2b640 1 -- 172.21.15.138:0/4165282126 wait complete. 2024-03-29T11:29:03.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:03.423+0000 7f86a1f2b640 1 librados: shutdown 2024-03-29T11:29:03.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:10.237 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.393+0000 7f8173519640 10 monclient: get_monmap_and_config 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.393+0000 7f8173519640 10 monclient: build_initial_monmap 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.393+0000 7f8173519640 10 monclient: monmap: 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:10.395015+0000 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:10.395015+0000 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:10.385 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:10.386 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:10.386 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.397+0000 7f8173519640 1 Processor -- start 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.397+0000 7f8173519640 1 -- start start 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.397+0000 7f8173519640 10 monclient: init 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 10 monclient(hunting): picked mon.noname-c con 0x7f816c0b4920 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 10 monclient(hunting): picked mon.noname-b con 0x7f816c0b4200 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 10 monclient(hunting): picked mon.noname-a con 0x7f816c13f9a0 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0b0930 con 0x7f816c13f9a0 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0af460 con 0x7f816c0b4920 2024-03-29T11:29:10.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0aeda0 con 0x7f816c0b4200 2024-03-29T11:29:10.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:10.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.399+0000 7f8173519640 10 monclient(hunting): authenticate will time out at 1371.708200s 2024-03-29T11:29:10.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.402+0000 7f8171d16640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f816c0b4200 0x7f816c0b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:33676/0 (socket says 172.21.15.138:33676) 2024-03-29T11:29:10.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.402+0000 7f8171d16640 1 -- 172.21.15.138:0/2602763160 learned_addr learned my addr 172.21.15.138:0/2602763160 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152004609 0 0) 0x7f816c0b0930 con 0x7f816c13f9a0 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8158003520 con 0x7f816c13f9a0 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013069357 0 0) 0x7f816c0af460 con 0x7f816c0b4920 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c0b0930 con 0x7f816c0b4920 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 417970901 0 0) 0x7f8158003520 con 0x7f816c13f9a0 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 >> v1:172.21.15.138:6790/0 conn(0x7f816c0b4200 legacy=0x7f816c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 >> v1:172.21.15.152:6789/0 conn(0x7f816c0b4920 legacy=0x7f816c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: _finish_hunting 0 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 monclient: found mon.noname-a 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0af980 con 0x7f816c13f9a0 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: _finish_auth 0 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: _check_auth_tickets 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:10.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8164001a10 con 0x7f816c13f9a0 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: dump: 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8173519640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:29:10.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8173519640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8164001c30 con 0x7f816c13f9a0 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 1 -- 172.21.15.138:0/2602763160 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8164001fc0 con 0x7f816c13f9a0 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8171515640 10 monclient: dump: 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.398 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:10.399 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:10.399 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:10.399 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:10.399 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:10.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8173519640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:10.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.403+0000 7f8173519640 4 set_mon_vals no callback set 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.404+0000 7f8173519640 10 monclient: shutdown 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.404+0000 7f8173519640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.404+0000 7f8173519640 1 -- 172.21.15.138:0/2602763160 >> v1:172.21.15.138:6789/0 conn(0x7f816c13f9a0 legacy=0x7f816c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.406+0000 7f8173519640 1 -- 172.21.15.138:0/2602763160 shutdown_connections 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.406+0000 7f8173519640 1 -- 172.21.15.138:0/2602763160 >> 172.21.15.138:0/2602763160 conn(0x7f816c06d230 msgr2=0x7f816c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.406+0000 7f8173519640 1 -- 172.21.15.138:0/2602763160 shutdown_connections 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.406+0000 7f8173519640 1 -- 172.21.15.138:0/2602763160 wait complete. 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.407+0000 7f8173519640 10 monclient: build_initial_monmap 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.407+0000 7f8173519640 10 monclient: monmap: 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:10.402 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.408+0000 7f8173519640 1 librados: starting msgr at 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.408+0000 7f8173519640 1 librados: starting objecter 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.408+0000 7f8173519640 1 Processor -- start 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.408+0000 7f8173519640 1 -- start start 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.408+0000 7f8173519640 1 librados: setting wanted keys 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.408+0000 7f8173519640 1 librados: calling monclient init 2024-03-29T11:29:10.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.408+0000 7f8173519640 10 monclient: init 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.409+0000 7f8173519640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.409+0000 7f8173519640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.410+0000 7f8173519640 10 monclient(hunting): picked mon.noname-b con 0x7f816c0b4200 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.410+0000 7f8173519640 10 monclient(hunting): picked mon.noname-a con 0x7f816c0b4920 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8173519640 10 monclient(hunting): picked mon.noname-c con 0x7f816c13f9a0 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8173519640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c071220 con 0x7f816c0b4920 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8173519640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c13c4c0 con 0x7f816c0b4200 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8173519640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c13c7a0 con 0x7f816c13f9a0 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8173519640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:10.404 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8173519640 10 monclient(hunting): authenticate will time out at 1371.720275s 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8172d18640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f816c13f9a0 0x7f816c1e6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:38718/0 (socket says 172.21.15.138:38718) 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8172d18640 1 -- 172.21.15.138:0/2793400604 learned_addr learned my addr 172.21.15.138:0/2793400604 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8162ffd640 10 client.?.objecter ms_handle_connect 0x7f816c13f9a0 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.411+0000 7f8162ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785630518 0 0) 0x7f816c13c7a0 con 0x7f816c13f9a0 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150004eb0 con 0x7f816c13f9a0 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 client.?.objecter ms_handle_connect 0x7f816c0b4200 2024-03-29T11:29:10.410 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694528888 0 0) 0x7f8150004eb0 con 0x7f816c13f9a0 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f816c13c7a0 con 0x7f816c13f9a0 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8164001f40 con 0x7f816c13f9a0 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:10.411 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895466991 0 0) 0x7f816c13c4c0 con 0x7f816c0b4200 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150004eb0 con 0x7f816c0b4200 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.412+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3456413949 0 0) 0x7f816c13c7a0 con 0x7f816c13f9a0 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.413+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 >> v1:172.21.15.152:6789/0 conn(0x7f816c0b4200 legacy=0x7f816c13a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.413+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 >> v1:172.21.15.138:6789/0 conn(0x7f816c0b4920 legacy=0x7f816c06e070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.416+0000 7f8162ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:10.412 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.416+0000 7f8162ffd640 1 monclient: found mon.c 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.416+0000 7f8162ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.416+0000 7f8162ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.416+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c13b620 con 0x7f816c13f9a0 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.416+0000 7f8162ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.416+0000 7f8162ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 10 monclient: _renew_subs 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c138d20 con 0x7f816c13f9a0 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 10 monclient: _renew_subs 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:10.413 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c136440 con 0x7f816c13f9a0 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 10 client.4598.objecter ms_handle_connect 0x7f816c0b4920 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8173519640 1 librados: init done 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 10 client.4598.objecter resend_mon_ops 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8164003c20 con 0x7f816c13f9a0 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f8164004070 con 0x7f816c13f9a0 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8162ffd640 10 monclient: dump: 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:10.414 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.415 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:10.415 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:10.415 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:10.415 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:10.415 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:10.415 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.417+0000 7f8170d14640 4 set_mon_vals no callback set 2024-03-29T11:29:10.416 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.419+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f81640041d0 con 0x7f816c13f9a0 2024-03-29T11:29:10.416 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.420+0000 7f8172517640 10 monclient: get_auth_request con 0x7f815006dc90 auth_method 0 2024-03-29T11:29:10.416 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.420+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f8164084eb0 con 0x7f816c13f9a0 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.420+0000 7f8162ffd640 10 client.4598.objecter ms_dispatch 0x7f816c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.420+0000 7f8162ffd640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.420+0000 7f8162ffd640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.420+0000 7f8162ffd640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.420+0000 7f8162ffd640 10 client.4598.objecter ms_handle_connect 0x7f815006dc90 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.421+0000 7f8173519640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.421+0000 7f8173519640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.421+0000 7f8173519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:10.417 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.421+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8154005080 con 0x7f816c13f9a0 2024-03-29T11:29:10.420 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.429+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8164050790 con 0x7f816c13f9a0 2024-03-29T11:29:10.420 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.429+0000 7f8162ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.420 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.429+0000 7f8162ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:10.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.759+0000 7f8173519640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.759+0000 7f8173519640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.750 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.759+0000 7f8173519640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:10.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.759+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8154005330 con 0x7f816c13f9a0 2024-03-29T11:29:10.751 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:10.751 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:10.751 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.760+0000 7f8162ffd640 1 -- 172.21.15.138:0/2793400604 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f81640526c0 con 0x7f816c13f9a0 2024-03-29T11:29:10.752 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.760+0000 7f8162ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.752 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.760+0000 7f8162ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:10.754 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.763+0000 7f8173519640 10 librados: watch_flush enter 2024-03-29T11:29:10.755 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 10 librados: watch_flush exit 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 >> v1:172.21.15.152:6816/3052585572 conn(0x7f815006dc90 legacy=0x7f8150070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 10 monclient: shutdown 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 >> v1:172.21.15.138:6790/0 conn(0x7f816c13f9a0 legacy=0x7f816c1e6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 shutdown_connections 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.764+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 >> 172.21.15.138:0/2793400604 conn(0x7f816c06d230 msgr2=0x7f816c0b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.765+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 shutdown_connections 2024-03-29T11:29:10.756 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.765+0000 7f8173519640 1 -- 172.21.15.138:0/2793400604 wait complete. 2024-03-29T11:29:10.757 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:10.765+0000 7f8173519640 1 librados: shutdown 2024-03-29T11:29:10.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:17.577 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.727+0000 7f9bd054b640 10 monclient: get_monmap_and_config 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.727+0000 7f9bd054b640 10 monclient: build_initial_monmap 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.727+0000 7f9bd054b640 10 monclient: monmap: 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:17.729143+0000 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:17.729143+0000 2024-03-29T11:29:17.719 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.728+0000 7f9bd054b640 1 Processor -- start 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.728+0000 7f9bd054b640 1 -- start start 2024-03-29T11:29:17.720 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.728+0000 7f9bd054b640 10 monclient: init 2024-03-29T11:29:17.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.729+0000 7f9bd054b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:17.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.729+0000 7f9bd054b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:17.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc80b4790 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:17.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 10 monclient(hunting): picked mon.noname-b con 0x7f9bc80aea90 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:17.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc80b39f0 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:17.721 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc80b0c60 con 0x7f9bc80b39f0 2024-03-29T11:29:17.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc80af020 con 0x7f9bc80b4790 2024-03-29T11:29:17.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc813fbd0 con 0x7f9bc80aea90 2024-03-29T11:29:17.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:17.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bd054b640 10 monclient(hunting): authenticate will time out at 1379.039240s 2024-03-29T11:29:17.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bceae8640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f9bc80b39f0 0x7f9bc813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34268/0 (socket says 172.21.15.138:34268) 2024-03-29T11:29:17.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bceae8640 1 -- 172.21.15.138:0/2486964112 learned_addr learned my addr 172.21.15.138:0/2486964112 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:17.722 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.730+0000 7f9bcdae6640 1 --1- 172.21.15.138:0/2486964112 >> v1:172.21.15.138:6790/0 conn(0x7f9bc80aea90 0x7f9bc80af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:57452/0 (socket says 172.21.15.138:57452) 2024-03-29T11:29:17.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197195752 0 0) 0x7f9bc80b0c60 con 0x7f9bc80b39f0 2024-03-29T11:29:17.724 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb4003520 con 0x7f9bc80b39f0 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3102514016 0 0) 0x7f9bb4003520 con 0x7f9bc80b39f0 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 >> v1:172.21.15.138:6790/0 conn(0x7f9bc80aea90 legacy=0x7f9bc80af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 >> v1:172.21.15.152:6789/0 conn(0x7f9bc80b4790 legacy=0x7f9bc80b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: _finish_hunting 0 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 monclient: found mon.noname-a 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc813fd60 con 0x7f9bc80b39f0 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: _finish_auth 0 2024-03-29T11:29:17.725 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: _check_auth_tickets 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9bc0001a10 con 0x7f9bc80b39f0 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: dump: 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:17.726 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bd054b640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bd054b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bc0001be0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 1 -- 172.21.15.138:0/2486964112 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9bc0002030 con 0x7f9bc80b39f0 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.731+0000 7f9bcd2e5640 10 monclient: dump: 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.727 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.732+0000 7f9bd054b640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.732+0000 7f9bd054b640 4 set_mon_vals no callback set 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.732+0000 7f9bd054b640 10 monclient: shutdown 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.732+0000 7f9bd054b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.732+0000 7f9bd054b640 1 -- 172.21.15.138:0/2486964112 >> v1:172.21.15.138:6789/0 conn(0x7f9bc80b39f0 legacy=0x7f9bc813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.733+0000 7f9bd054b640 1 -- 172.21.15.138:0/2486964112 shutdown_connections 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.733+0000 7f9bd054b640 1 -- 172.21.15.138:0/2486964112 >> 172.21.15.138:0/2486964112 conn(0x7f9bc8066a00 msgr2=0x7f9bc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.733+0000 7f9bd054b640 1 -- 172.21.15.138:0/2486964112 shutdown_connections 2024-03-29T11:29:17.728 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.733+0000 7f9bd054b640 1 -- 172.21.15.138:0/2486964112 wait complete. 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.734+0000 7f9bd054b640 10 monclient: build_initial_monmap 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.734+0000 7f9bd054b640 10 monclient: monmap: 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:17.734 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.734+0000 7f9bd054b640 1 librados: starting msgr at 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.735+0000 7f9bd054b640 1 librados: starting objecter 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.735+0000 7f9bd054b640 1 Processor -- start 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.735+0000 7f9bd054b640 1 -- start start 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.735+0000 7f9bd054b640 1 librados: setting wanted keys 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.735+0000 7f9bd054b640 1 librados: calling monclient init 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.735+0000 7f9bd054b640 10 monclient: init 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc80b39f0 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bce2e7640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f9bc80b39f0 0x7f9bc8142380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:57972/0 (socket says 172.21.15.138:57972) 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bce2e7640 1 -- 172.21.15.138:0/4012420155 learned_addr learned my addr 172.21.15.138:0/4012420155 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:17.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 10 monclient(hunting): picked mon.noname-b con 0x7f9bc80b4790 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc8134b80 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc8140030 con 0x7f9bc8134b80 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc8142a90 con 0x7f9bc80b4790 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc8142c20 con 0x7f9bc80b39f0 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bbeffd640 10 client.?.objecter ms_handle_connect 0x7f9bc80b39f0 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.736+0000 7f9bd054b640 10 monclient(hunting): authenticate will time out at 1379.045852s 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 client.?.objecter ms_handle_connect 0x7f9bc80b4790 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819809461 0 0) 0x7f9bc8142c20 con 0x7f9bc80b39f0 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac004eb0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070089849 0 0) 0x7f9bc8142a90 con 0x7f9bc80b4790 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc8142c20 con 0x7f9bc80b4790 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 client.?.objecter ms_handle_connect 0x7f9bc8134b80 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3900457171 0 0) 0x7f9bac004eb0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bc8142a90 con 0x7f9bc80b39f0 2024-03-29T11:29:17.737 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9bc40017d0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): dump: 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:17.738 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479193031 0 0) 0x7f9bc8140030 con 0x7f9bc8134b80 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.737+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac004eb0 con 0x7f9bc8134b80 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3244617886 0 0) 0x7f9bc8142c20 con 0x7f9bc80b4790 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bc8140030 con 0x7f9bc80b4790 2024-03-29T11:29:17.739 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9bb8001930 con 0x7f9bc80b4790 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 10 monclient(hunting): dump: 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:17.740 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3673527746 0 0) 0x7f9bc8142a90 con 0x7f9bc80b39f0 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 >> v1:172.21.15.152:6789/0 conn(0x7f9bc80b4790 legacy=0x7f9bc81404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 >> v1:172.21.15.138:6789/0 conn(0x7f9bc8134b80 legacy=0x7f9bc8140bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 monclient: found mon.c 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8142db0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 10 monclient: _renew_subs 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:17.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc81e3470 con 0x7f9bc80b39f0 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 10 monclient: _renew_subs 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc81e37e0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.738+0000 7f9bd054b640 1 librados: init done 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.739+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bc4001ce0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.739+0000 7f9bbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.739+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9bc4004a00 con 0x7f9bc80b39f0 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.739+0000 7f9bbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.739+0000 7f9bbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.739+0000 7f9bbeffd640 10 monclient: dump: 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:17.742 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.739+0000 7f9bccae4640 4 set_mon_vals no callback set 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.740+0000 7f9bd054b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.740+0000 7f9bd054b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.740+0000 7f9bd054b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.740+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb0005080 con 0x7f9bc80b39f0 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.741+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f9bc401a9d0 con 0x7f9bc80b39f0 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.742+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f9bc4084910 con 0x7f9bc80b39f0 2024-03-29T11:29:17.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.742+0000 7f9bbeffd640 10 client.4604.objecter ms_dispatch 0x7f9bc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:17.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.742+0000 7f9bbeffd640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:17.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.742+0000 7f9bbeffd640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:17.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.742+0000 7f9bbeffd640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:29:17.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.742+0000 7f9bcdae6640 10 monclient: get_auth_request con 0x7f9bac06e000 auth_method 0 2024-03-29T11:29:17.744 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.742+0000 7f9bbeffd640 10 client.4604.objecter ms_handle_connect 0x7f9bac06e000 2024-03-29T11:29:17.746 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.755+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9bc4050020 con 0x7f9bc80b39f0 2024-03-29T11:29:17.747 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.755+0000 7f9bbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.747 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:17.755+0000 7f9bbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:18.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.122+0000 7f9bd054b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.122+0000 7f9bd054b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.122+0000 7f9bd054b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:18.114 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.122+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bb0005330 con 0x7f9bc80b39f0 2024-03-29T11:29:18.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.124+0000 7f9bbeffd640 1 -- 172.21.15.138:0/4012420155 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f9bc4052120 con 0x7f9bc80b39f0 2024-03-29T11:29:18.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.124+0000 7f9bbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.115 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.124+0000 7f9bbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:18.115 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:18.116 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:18.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.127+0000 7f9bd054b640 10 librados: watch_flush enter 2024-03-29T11:29:18.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.127+0000 7f9bd054b640 10 librados: watch_flush exit 2024-03-29T11:29:18.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.127+0000 7f9bd054b640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:29:18.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.127+0000 7f9bd054b640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:29:18.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.128+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 >> v1:172.21.15.152:6816/3052585572 conn(0x7f9bac06e000 legacy=0x7f9bac0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.128+0000 7f9bd054b640 10 monclient: shutdown 2024-03-29T11:29:18.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.128+0000 7f9bd054b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:18.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.128+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 >> v1:172.21.15.138:6790/0 conn(0x7f9bc80b39f0 legacy=0x7f9bc8142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.128+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 shutdown_connections 2024-03-29T11:29:18.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.128+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 >> 172.21.15.138:0/4012420155 conn(0x7f9bc8066a00 msgr2=0x7f9bc80b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:18.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.129+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 shutdown_connections 2024-03-29T11:29:18.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.129+0000 7f9bd054b640 1 -- 172.21.15.138:0/4012420155 wait complete. 2024-03-29T11:29:18.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:18.129+0000 7f9bd054b640 1 librados: shutdown 2024-03-29T11:29:18.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:24.943 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.103+0000 7fc9a6693640 10 monclient: get_monmap_and_config 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.103+0000 7fc9a6693640 10 monclient: build_initial_monmap 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.103+0000 7fc9a6693640 10 monclient: monmap: 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:25.104671+0000 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:25.104671+0000 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:25.095 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:25.096 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:25.096 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:25.096 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.104+0000 7fc9a6693640 1 Processor -- start 2024-03-29T11:29:25.096 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.104+0000 7fc9a6693640 1 -- start start 2024-03-29T11:29:25.096 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.104+0000 7fc9a6693640 10 monclient: init 2024-03-29T11:29:25.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:25.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:25.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a006ade0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:25.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a006ed60 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:25.099 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a0072340 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:25.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a006ff50 con 0x7fc9a0072340 2024-03-29T11:29:25.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a0070180 con 0x7fc9a006ade0 2024-03-29T11:29:25.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a006f310 con 0x7fc9a006ed60 2024-03-29T11:29:25.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:25.100 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.105+0000 7fc9a6693640 10 monclient(hunting): authenticate will time out at 1386.414741s 2024-03-29T11:29:25.101 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.109+0000 7fc99ffff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fc9a006ade0 0x7fc9a01499e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:52231/0 (socket says 172.21.15.138:52231) 2024-03-29T11:29:25.101 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.109+0000 7fc99ffff640 1 -- 172.21.15.138:0/1880205289 learned_addr learned my addr 172.21.15.138:0/1880205289 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:25.101 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.110+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034749470 0 0) 0x7fc9a0070180 con 0x7fc9a006ade0 2024-03-29T11:29:25.101 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.110+0000 7fc99f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:25.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.110+0000 7fc99f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:25.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.110+0000 7fc99f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:25.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.110+0000 7fc99f7fe640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:29:25.102 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.110+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc98c003520 con 0x7fc9a006ade0 2024-03-29T11:29:25.107 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4265137428 0 0) 0x7fc98c003520 con 0x7fc9a006ade0 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 >> v1:172.21.15.138:6790/0 conn(0x7fc9a006ed60 legacy=0x7fc9a0071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 >> v1:172.21.15.138:6789/0 conn(0x7fc9a0072340 legacy=0x7fc9a0151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a00aeed0 con 0x7fc9a006ade0 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc99f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc9a6693640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.112+0000 7fc9a6693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:25.108 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc990001690 con 0x7fc9a006ade0 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc99f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc99f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc99f7fe640 10 monclient: dump: 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc9a6693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:25.109 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc99f7fe640 1 -- 172.21.15.138:0/1880205289 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9900018b0 con 0x7fc9a006ade0 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc99f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc9a6693640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.113+0000 7fc9a6693640 4 set_mon_vals no callback set 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.114+0000 7fc9a6693640 10 monclient: shutdown 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.114+0000 7fc9a6693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.114+0000 7fc9a6693640 1 -- 172.21.15.138:0/1880205289 >> v1:172.21.15.152:6789/0 conn(0x7fc9a006ade0 legacy=0x7fc9a01499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.114+0000 7fc9a6693640 1 -- 172.21.15.138:0/1880205289 shutdown_connections 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.114+0000 7fc9a6693640 1 -- 172.21.15.138:0/1880205289 >> 172.21.15.138:0/1880205289 conn(0x7fc9a00b4930 msgr2=0x7fc9a00b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.115+0000 7fc9a6693640 1 -- 172.21.15.138:0/1880205289 shutdown_connections 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.115+0000 7fc9a6693640 1 -- 172.21.15.138:0/1880205289 wait complete. 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.115+0000 7fc9a6693640 10 monclient: build_initial_monmap 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.115+0000 7fc9a6693640 10 monclient: monmap: 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:25.110 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.116+0000 7fc9a6693640 1 librados: starting msgr at 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.116+0000 7fc9a6693640 1 librados: starting objecter 2024-03-29T11:29:25.111 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.116+0000 7fc9a6693640 1 Processor -- start 2024-03-29T11:29:25.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.116+0000 7fc9a6693640 1 -- start start 2024-03-29T11:29:25.112 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.116+0000 7fc9a6693640 1 librados: setting wanted keys 2024-03-29T11:29:25.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.116+0000 7fc9a6693640 1 librados: calling monclient init 2024-03-29T11:29:25.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.116+0000 7fc9a6693640 10 monclient: init 2024-03-29T11:29:25.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.117+0000 7fc9a6693640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:25.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.117+0000 7fc9a6693640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:25.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.117+0000 7fc9a6693640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a0072340 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:25.113 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.117+0000 7fc9a6693640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a00a9d70 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:25.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.119+0000 7fc9a6693640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a0071280 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:25.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.119+0000 7fc9a6693640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a00ab290 con 0x7fc9a00a9d70 2024-03-29T11:29:25.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.119+0000 7fc9a6693640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a01e4a80 con 0x7fc9a0072340 2024-03-29T11:29:25.118 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.119+0000 7fc9a6693640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a00ab500 con 0x7fc9a0071280 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.119+0000 7fc9a6693640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.119+0000 7fc9a6693640 10 monclient(hunting): authenticate will time out at 1386.428348s 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.121+0000 7fc99ffff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fc9a0072340 0x7fc9a01e7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:52295/0 (socket says 172.21.15.138:52295) 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.121+0000 7fc99ffff640 1 -- 172.21.15.138:0/4124960893 learned_addr learned my addr 172.21.15.138:0/4124960893 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 client.?.objecter ms_handle_connect 0x7fc9a0072340 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989103915 0 0) 0x7fc9a01e4a80 con 0x7fc9a0072340 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc980004eb0 con 0x7fc9a0072340 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2613575875 0 0) 0x7fc980004eb0 con 0x7fc9a0072340 2024-03-29T11:29:25.119 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a01e4a80 con 0x7fc9a0072340 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc9900047e0 con 0x7fc9a0072340 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): dump: 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:25.120 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1085130674 0 0) 0x7fc9a01e4a80 con 0x7fc9a0072340 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 >> v1:172.21.15.138:6790/0 conn(0x7fc9a0071280 legacy=0x7fc9a01e4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.122+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 >> v1:172.21.15.138:6789/0 conn(0x7fc9a00a9d70 legacy=0x7fc9a01e3c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.123+0000 7fc99d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.123+0000 7fc99d7fa640 1 monclient: found mon.b 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.123+0000 7fc99d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.123+0000 7fc99d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.123+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a00ab7e0 con 0x7fc9a0072340 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 10 monclient: _renew_subs 2024-03-29T11:29:25.121 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a00a9850 con 0x7fc9a0072340 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 10 monclient: _renew_subs 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a00a94e0 con 0x7fc9a0072340 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc9a6693640 1 librados: init done 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc990001aa0 con 0x7fc9a0072340 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fc990004dd0 con 0x7fc9a0072340 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.124+0000 7fc99d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.125+0000 7fc99d7fa640 10 monclient: dump: 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:25.122 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.126+0000 7fc99effd640 4 set_mon_vals no callback set 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.126+0000 7fc9a6693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.123 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.126+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fc990004f30 con 0x7fc9a0072340 2024-03-29T11:29:25.124 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.127+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fc990084af0 con 0x7fc9a0072340 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.127+0000 7fc99d7fa640 10 client.4498.objecter ms_dispatch 0x7fc9a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.127+0000 7fc99d7fa640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.127+0000 7fc99d7fa640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.127+0000 7fc99d7fa640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.130+0000 7fc9a6693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.130+0000 7fc9a6693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.130+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a006b1c0 con 0x7fc9a0072340 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.130+0000 7fc997fff640 10 monclient: get_auth_request con 0x7fc98006dd60 auth_method 0 2024-03-29T11:29:25.125 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.130+0000 7fc99d7fa640 10 client.4498.objecter ms_handle_connect 0x7fc98006dd60 2024-03-29T11:29:25.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.140+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9900503d0 con 0x7fc9a0072340 2024-03-29T11:29:25.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.141+0000 7fc99d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.132 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.141+0000 7fc99d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:25.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.497+0000 7fc9a6693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.497+0000 7fc9a6693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.497+0000 7fc9a6693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:25.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.497+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9a0075f10 con 0x7fc9a0072340 2024-03-29T11:29:25.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.499+0000 7fc99d7fa640 1 -- 172.21.15.138:0/4124960893 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fc990052300 con 0x7fc9a0072340 2024-03-29T11:29:25.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.499+0000 7fc99d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.499+0000 7fc99d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:25.491 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:25.491 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 10 librados: watch_flush enter 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 10 librados: watch_flush exit 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 10 client.4498.objecter successfully canceled tick 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 >> v1:172.21.15.152:6816/3052585572 conn(0x7fc98006dd60 legacy=0x7fc980070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 10 monclient: shutdown 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 >> v1:172.21.15.152:6789/0 conn(0x7fc9a0072340 legacy=0x7fc9a01e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 shutdown_connections 2024-03-29T11:29:25.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 >> 172.21.15.138:0/4124960893 conn(0x7fc9a00b4930 msgr2=0x7fc9a00afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:25.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 shutdown_connections 2024-03-29T11:29:25.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.503+0000 7fc9a6693640 1 -- 172.21.15.138:0/4124960893 wait complete. 2024-03-29T11:29:25.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:25.504+0000 7fc9a6693640 1 librados: shutdown 2024-03-29T11:29:25.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:32.318 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:32.471 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.480+0000 7f5a5b5f4640 10 monclient: get_monmap_and_config 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.480+0000 7f5a5b5f4640 10 monclient: build_initial_monmap 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.480+0000 7f5a5b5f4640 10 monclient: monmap: 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:32.481301+0000 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:32.481301+0000 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:32.472 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.481+0000 7f5a5b5f4640 1 Processor -- start 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 1 -- start start 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 10 monclient: init 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f5a540ae9b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5a54143f50 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:32.475 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5a540b4890 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:32.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.482+0000 7f5a5b5f4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a540aed90 con 0x7f5a540b4890 2024-03-29T11:29:32.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.483+0000 7f5a5b5f4640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a540af020 con 0x7f5a540ae9b0 2024-03-29T11:29:32.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.483+0000 7f5a5b5f4640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a540af300 con 0x7f5a54143f50 2024-03-29T11:29:32.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.483+0000 7f5a5b5f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:32.476 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.483+0000 7f5a5b5f4640 10 monclient(hunting): authenticate will time out at 1393.792076s 2024-03-29T11:29:32.477 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a58b8f640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f5a54143f50 0x7f5a540b4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:48647/0 (socket says 172.21.15.138:48647) 2024-03-29T11:29:32.477 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a58b8f640 1 -- 172.21.15.138:0/1154136674 learned_addr learned my addr 172.21.15.138:0/1154136674 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039321151 0 0) 0x7f5a540af300 con 0x7f5a54143f50 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a4bfff640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.486+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a40003520 con 0x7f5a54143f50 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.487+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1083781157 0 0) 0x7f5a40003520 con 0x7f5a54143f50 2024-03-29T11:29:32.479 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.487+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 >> v1:172.21.15.152:6789/0 conn(0x7f5a540ae9b0 legacy=0x7f5a54143840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:32.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 >> v1:172.21.15.138:6789/0 conn(0x7f5a540b4890 legacy=0x7f5a54151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:32.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 1 monclient: found mon.noname-b 2024-03-29T11:29:32.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:32.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:29:32.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a540b0710 con 0x7f5a54143f50 2024-03-29T11:29:32.482 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 10 monclient: _finish_auth 0 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5a4c001a10 con 0x7f5a54143f50 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a4bfff640 10 monclient: dump: 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:32.483 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:32.484 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a5b5f4640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:29:32.484 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.489+0000 7f5a5b5f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:32.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.492+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a4c001c70 con 0x7f5a54143f50 2024-03-29T11:29:32.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.492+0000 7f5a4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:32.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.492+0000 7f5a4bfff640 1 -- 172.21.15.138:0/1154136674 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5a4c002000 con 0x7f5a54143f50 2024-03-29T11:29:32.485 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.492+0000 7f5a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.492+0000 7f5a4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.492+0000 7f5a4bfff640 10 monclient: dump: 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.486 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:32.487 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:32.487 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:32.487 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.493+0000 7f5a5b5f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:32.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.493+0000 7f5a5b5f4640 4 set_mon_vals no callback set 2024-03-29T11:29:32.487 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.493+0000 7f5a5b5f4640 10 monclient: shutdown 2024-03-29T11:29:32.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.493+0000 7f5a5b5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:32.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.493+0000 7f5a5b5f4640 1 -- 172.21.15.138:0/1154136674 >> v1:172.21.15.138:6790/0 conn(0x7f5a54143f50 legacy=0x7f5a540b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.494+0000 7f5a5b5f4640 1 -- 172.21.15.138:0/1154136674 shutdown_connections 2024-03-29T11:29:32.488 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.494+0000 7f5a5b5f4640 1 -- 172.21.15.138:0/1154136674 >> 172.21.15.138:0/1154136674 conn(0x7f5a54066a00 msgr2=0x7f5a54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.494+0000 7f5a5b5f4640 1 -- 172.21.15.138:0/1154136674 shutdown_connections 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.494+0000 7f5a5b5f4640 1 -- 172.21.15.138:0/1154136674 wait complete. 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.495+0000 7f5a5b5f4640 10 monclient: build_initial_monmap 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.495+0000 7f5a5b5f4640 10 monclient: monmap: 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:32.489 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.495+0000 7f5a5b5f4640 1 librados: starting msgr at 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.495+0000 7f5a5b5f4640 1 librados: starting objecter 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.495+0000 7f5a5b5f4640 1 Processor -- start 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.496+0000 7f5a5b5f4640 1 -- start start 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.496+0000 7f5a5b5f4640 1 librados: setting wanted keys 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.496+0000 7f5a5b5f4640 1 librados: calling monclient init 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.496+0000 7f5a5b5f4640 10 monclient: init 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.496+0000 7f5a5b5f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.496+0000 7f5a5b5f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.496+0000 7f5a5b5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5a540ae9b0 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:32.490 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.497+0000 7f5a5b5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f5a540b4890 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:32.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.497+0000 7f5a5b5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5a54143f50 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:32.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.497+0000 7f5a5b5f4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a540a9e60 con 0x7f5a540ae9b0 2024-03-29T11:29:32.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.497+0000 7f5a5b5f4640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a540ab100 con 0x7f5a54143f50 2024-03-29T11:29:32.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.497+0000 7f5a5b5f4640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a540ab3e0 con 0x7f5a540b4890 2024-03-29T11:29:32.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.497+0000 7f5a5b5f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:32.491 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.497+0000 7f5a5b5f4640 10 monclient(hunting): authenticate will time out at 1393.806118s 2024-03-29T11:29:32.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.502+0000 7f5a59b91640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f5a54143f50 0x7f5a5406f420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:41447/0 (socket says 172.21.15.138:41447) 2024-03-29T11:29:32.494 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.502+0000 7f5a59b91640 1 -- 172.21.15.138:0/3734931835 learned_addr learned my addr 172.21.15.138:0/3734931835 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:32.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 client.?.objecter ms_handle_connect 0x7f5a540b4890 2024-03-29T11:29:32.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 client.?.objecter ms_handle_connect 0x7f5a540ae9b0 2024-03-29T11:29:32.495 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.496 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590984466 0 0) 0x7f5a540ab3e0 con 0x7f5a540b4890 2024-03-29T11:29:32.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.497 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a38004eb0 con 0x7f5a540b4890 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 client.?.objecter ms_handle_connect 0x7f5a54143f50 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.503+0000 7f5a49ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802373073 0 0) 0x7f5a540a9e60 con 0x7f5a540ae9b0 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a380051d0 con 0x7f5a540ae9b0 2024-03-29T11:29:32.498 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966247560 0 0) 0x7f5a540ab100 con 0x7f5a54143f50 2024-03-29T11:29:32.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:29:32.499 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.504+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a38005510 con 0x7f5a54143f50 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.505+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992445617 0 0) 0x7f5a38004eb0 con 0x7f5a540b4890 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.505+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a540ab100 con 0x7f5a540b4890 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.505+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2992149446 0 0) 0x7f5a380051d0 con 0x7f5a540ae9b0 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.505+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a38004eb0 con 0x7f5a540ae9b0 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5a4c001f80 con 0x7f5a540b4890 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:32.500 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5a50001790 con 0x7f5a540ae9b0 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:32.501 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.502 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996151657 0 0) 0x7f5a38005510 con 0x7f5a54143f50 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a380051d0 con 0x7f5a54143f50 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5a44001850 con 0x7f5a54143f50 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.508+0000 7f5a49ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:32.503 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:32.504 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2032383260 0 0) 0x7f5a380051d0 con 0x7f5a54143f50 2024-03-29T11:29:32.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 >> v1:172.21.15.138:6790/0 conn(0x7f5a540b4890 legacy=0x7f5a5413b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 >> v1:172.21.15.138:6789/0 conn(0x7f5a540ae9b0 legacy=0x7f5a540a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.504 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 1 monclient: found mon.b 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a54071a20 con 0x7f5a54143f50 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 10 monclient: _renew_subs 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5413bba0 con 0x7f5a54143f50 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 10 monclient: _renew_subs 2024-03-29T11:29:32.505 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:32.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5406fb30 con 0x7f5a54143f50 2024-03-29T11:29:32.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.509+0000 7f5a5b5f4640 1 librados: init done 2024-03-29T11:29:32.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.511+0000 7f5a3f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.511+0000 7f5a3f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.511+0000 7f5a3f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:32.506 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.511+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a34005080 con 0x7f5a54143f50 2024-03-29T11:29:32.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.513+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a44001d40 con 0x7f5a54143f50 2024-03-29T11:29:32.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.513+0000 7f5a49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:32.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.513+0000 7f5a4b7fe640 4 set_mon_vals no callback set 2024-03-29T11:29:32.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.514+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5a44001850 con 0x7f5a54143f50 2024-03-29T11:29:32.507 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.514+0000 7f5a49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.514+0000 7f5a49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.514+0000 7f5a49ffb640 10 monclient: dump: 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:32.508 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:32.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.515+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f5a44003670 con 0x7f5a54143f50 2024-03-29T11:29:32.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.516+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f5a44001670 con 0x7f5a54143f50 2024-03-29T11:29:32.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.516+0000 7f5a49ffb640 10 client.4501.objecter ms_dispatch 0x7f5a54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:32.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.516+0000 7f5a49ffb640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:32.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.516+0000 7f5a49ffb640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:32.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.516+0000 7f5a49ffb640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:29:32.509 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.516+0000 7f5a59390640 10 monclient: get_auth_request con 0x7f5a3806e8a0 auth_method 0 2024-03-29T11:29:32.510 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.519+0000 7f5a49ffb640 10 client.4501.objecter ms_handle_connect 0x7f5a3806e8a0 2024-03-29T11:29:32.518 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.526+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a44050410 con 0x7f5a54143f50 2024-03-29T11:29:32.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.526+0000 7f5a49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.519 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.526+0000 7f5a49ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:32.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.854+0000 7f5a3f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.854+0000 7f5a3f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.854+0000 7f5a3f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:32.846 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.854+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a34005330 con 0x7f5a54143f50 2024-03-29T11:29:32.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.858+0000 7f5a49ffb640 1 -- 172.21.15.138:0/3734931835 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f5a440505b0 con 0x7f5a54143f50 2024-03-29T11:29:32.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.858+0000 7f5a49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.858+0000 7f5a49ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:32.854 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:32.854 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:32.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.865+0000 7f5a3f7fe640 10 librados: watch_flush enter 2024-03-29T11:29:32.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.865+0000 7f5a3f7fe640 10 librados: watch_flush exit 2024-03-29T11:29:32.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.865+0000 7f5a3f7fe640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:29:32.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.865+0000 7f5a3f7fe640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:29:32.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.865+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 >> v1:172.21.15.152:6816/3052585572 conn(0x7f5a3806e8a0 legacy=0x7f5a38070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.866+0000 7f5a3f7fe640 10 monclient: shutdown 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.866+0000 7f5a3f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.866+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 >> v1:172.21.15.152:6789/0 conn(0x7f5a54143f50 legacy=0x7f5a5406f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.866+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 shutdown_connections 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.866+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 >> 172.21.15.138:0/3734931835 conn(0x7f5a54066a00 msgr2=0x7f5a54066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.866+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 shutdown_connections 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.867+0000 7f5a3f7fe640 1 -- 172.21.15.138:0/3734931835 wait complete. 2024-03-29T11:29:32.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:32.867+0000 7f5a3f7fe640 1 librados: shutdown 2024-03-29T11:29:32.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:39.682 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.852+0000 7facd16e6640 10 monclient: get_monmap_and_config 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.852+0000 7facd16e6640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.852+0000 7facd16e6640 10 monclient: monmap: 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:39.853299+0000 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:39.853299+0000 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:39.844 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:39.845 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:39.845 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:39.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.855+0000 7facd16e6640 1 Processor -- start 2024-03-29T11:29:39.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.855+0000 7facd16e6640 1 -- start start 2024-03-29T11:29:39.847 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.855+0000 7facd16e6640 10 monclient: init 2024-03-29T11:29:39.849 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 10 monclient(hunting): picked mon.noname-c con 0x7facc402b4f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 10 monclient(hunting): picked mon.noname-b con 0x7facc400b8d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 10 monclient(hunting): picked mon.noname-a con 0x7facc402a5d0 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc402c5f0 con 0x7facc402a5d0 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40c11d0 con 0x7facc402b4f0 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc402afd0 con 0x7facc400b8d0 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.856+0000 7facd16e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.857+0000 7facd16e6640 10 monclient(hunting): authenticate will time out at 1401.166056s 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.857+0000 7faccaffd640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7facc402b4f0 0x7facc40068c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:60650/0 (socket says 172.21.15.138:60650) 2024-03-29T11:29:39.850 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.857+0000 7faccaffd640 1 -- 172.21.15.138:0/3759619650 learned_addr learned my addr 172.21.15.138:0/3759619650 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:39.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.860+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701820891 0 0) 0x7facc40c11d0 con 0x7facc402b4f0 2024-03-29T11:29:39.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.860+0000 7facc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.860+0000 7facc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.860+0000 7facc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.860+0000 7facc9ffb640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:29:39.854 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.860+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb4003520 con 0x7facc402b4f0 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.862+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3774006123 0 0) 0x7facb4003520 con 0x7facc402b4f0 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 >> v1:172.21.15.138:6790/0 conn(0x7facc400b8d0 legacy=0x7facc40c0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 >> v1:172.21.15.138:6789/0 conn(0x7facc402a5d0 legacy=0x7facc40c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40cb190 con 0x7facc402b4f0 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.855 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faccc04f730 con 0x7facc402b4f0 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facd16e6640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facd16e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: dump: 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:39.856 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:39.857 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:39.857 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:39.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facd16e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:39.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faccc0008d0 con 0x7facc402b4f0 2024-03-29T11:29:39.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facd16e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:39.857 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.863+0000 7facd16e6640 4 set_mon_vals no callback set 2024-03-29T11:29:39.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facc9ffb640 1 -- 172.21.15.138:0/3759619650 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7faccc0758b0 con 0x7facc402b4f0 2024-03-29T11:29:39.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:39.858 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facc9ffb640 10 monclient: dump: 2024-03-29T11:29:39.858 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:39.858 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facd16e6640 10 monclient: shutdown 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facd16e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facd16e6640 1 -- 172.21.15.138:0/3759619650 >> v1:172.21.15.152:6789/0 conn(0x7facc402b4f0 legacy=0x7facc40068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facd16e6640 1 -- 172.21.15.138:0/3759619650 shutdown_connections 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.864+0000 7facd16e6640 1 -- 172.21.15.138:0/3759619650 >> 172.21.15.138:0/3759619650 conn(0x7facc4005920 msgr2=0x7facc4005d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:39.859 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.865+0000 7facd16e6640 1 -- 172.21.15.138:0/3759619650 shutdown_connections 2024-03-29T11:29:39.860 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.865+0000 7facd16e6640 1 -- 172.21.15.138:0/3759619650 wait complete. 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.866+0000 7facd16e6640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.866+0000 7facd16e6640 10 monclient: monmap: 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.866+0000 7facd16e6640 1 librados: starting msgr at 2024-03-29T11:29:39.861 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.866+0000 7facd16e6640 1 librados: starting objecter 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.867+0000 7facd16e6640 1 Processor -- start 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.867+0000 7facd16e6640 1 -- start start 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.867+0000 7facd16e6640 1 librados: setting wanted keys 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.867+0000 7facd16e6640 1 librados: calling monclient init 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.867+0000 7facd16e6640 10 monclient: init 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 10 monclient(hunting): picked mon.noname-a con 0x7facc400b8d0 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 10 monclient(hunting): picked mon.noname-c con 0x7facc402a5d0 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 10 monclient(hunting): picked mon.noname-b con 0x7facc402b4f0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40b2780 con 0x7facc400b8d0 2024-03-29T11:29:39.862 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40b45b0 con 0x7facc402b4f0 2024-03-29T11:29:39.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40c1820 con 0x7facc402a5d0 2024-03-29T11:29:39.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7facd16e6640 10 monclient(hunting): authenticate will time out at 1401.177474s 2024-03-29T11:29:39.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7faccb7fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7facc402b4f0 0x7facc40ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:44382/0 (socket says 172.21.15.138:44382) 2024-03-29T11:29:39.863 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.868+0000 7faccb7fe640 1 -- 172.21.15.138:0/1507988983 learned_addr learned my addr 172.21.15.138:0/1507988983 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:39.864 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 client.?.objecter ms_handle_connect 0x7facc402b4f0 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436209795 0 0) 0x7facc40b45b0 con 0x7facc402b4f0 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8004eb0 con 0x7facc402b4f0 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 588830030 0 0) 0x7facb8004eb0 con 0x7facc402b4f0 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc40b45b0 con 0x7facc402b4f0 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7facbc0018b0 con 0x7facc402b4f0 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): dump: 2024-03-29T11:29:39.865 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.870+0000 7facb37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.871+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1353309937 0 0) 0x7facc40b45b0 con 0x7facc402b4f0 2024-03-29T11:29:39.866 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.871+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 >> v1:172.21.15.138:6790/0 conn(0x7facc402a5d0 legacy=0x7facc40c9900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:39.868 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 >> v1:172.21.15.138:6789/0 conn(0x7facc400b8d0 legacy=0x7facc40c0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 1 monclient: found mon.b 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40c1b00 con 0x7facc402b4f0 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 10 monclient: _renew_subs 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc40b29d0 con 0x7facc402b4f0 2024-03-29T11:29:39.869 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 10 monclient: _renew_subs 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facc40b42b0 con 0x7facc402b4f0 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.874+0000 7facd16e6640 1 librados: init done 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.875+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facbc001c50 con 0x7facc402b4f0 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.875+0000 7facb37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.875+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7facbc003a40 con 0x7facc402b4f0 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.875+0000 7facb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.875+0000 7facb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.875+0000 7facb37fe640 10 monclient: dump: 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.870 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.875+0000 7facc8ff9640 4 set_mon_vals no callback set 2024-03-29T11:29:39.871 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.876+0000 7facd16e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.872 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.877+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7facbc003ba0 con 0x7facc402b4f0 2024-03-29T11:29:39.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7facbc084ab0 con 0x7facc402b4f0 2024-03-29T11:29:39.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facb37fe640 10 client.4507.objecter ms_dispatch 0x7faccc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facb37fe640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facb37fe640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facb37fe640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facd16e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facd16e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.878+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac005080 con 0x7facc402b4f0 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.879+0000 7faccaffd640 10 monclient: get_auth_request con 0x7facb806dc10 auth_method 0 2024-03-29T11:29:39.937 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.879+0000 7facb37fe640 10 client.4507.objecter ms_handle_connect 0x7facb806dc10 2024-03-29T11:29:39.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.885+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facbc050390 con 0x7facc402b4f0 2024-03-29T11:29:39.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.885+0000 7facb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.938 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:39.885+0000 7facb37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:40.232 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.240+0000 7facd16e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.232 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.240+0000 7facd16e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.232 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.240+0000 7facd16e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:29:40.233 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.240+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facac005330 con 0x7facc402b4f0 2024-03-29T11:29:40.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.243+0000 7facb37fe640 1 -- 172.21.15.138:0/1507988983 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7facbc0522c0 con 0x7facc402b4f0 2024-03-29T11:29:40.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.243+0000 7facb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.243+0000 7facb37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:40.236 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:40.236 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 10 librados: watch_flush enter 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 10 librados: watch_flush exit 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 >> v1:172.21.15.152:6816/3052585572 conn(0x7facb806dc10 legacy=0x7facb80700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 10 monclient: shutdown 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 >> v1:172.21.15.152:6789/0 conn(0x7facc402b4f0 legacy=0x7facc40ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 shutdown_connections 2024-03-29T11:29:40.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.248+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 >> 172.21.15.138:0/1507988983 conn(0x7facc4005920 msgr2=0x7facc4036a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:40.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.249+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 shutdown_connections 2024-03-29T11:29:40.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.249+0000 7facd16e6640 1 -- 172.21.15.138:0/1507988983 wait complete. 2024-03-29T11:29:40.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:40.249+0000 7facd16e6640 1 librados: shutdown 2024-03-29T11:29:40.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:47.067 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.219+0000 7fd0288ae640 10 monclient: get_monmap_and_config 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.219+0000 7fd0288ae640 10 monclient: build_initial_monmap 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.219+0000 7fd0288ae640 10 monclient: monmap: 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:47.220590+0000 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:47.220590+0000 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:47.212 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:47.213 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:47.213 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:47.213 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:47.213 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:47.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.220+0000 7fd0288ae640 1 Processor -- start 2024-03-29T11:29:47.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.220+0000 7fd0288ae640 1 -- start start 2024-03-29T11:29:47.213 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.220+0000 7fd0288ae640 10 monclient: init 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd0241422b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd024142690 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd0240b17d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd024142c80 con 0x7fd024142690 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240aae40 con 0x7fd0240b17d0 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02413fbd0 con 0x7fd0241422b0 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd0288ae640 10 monclient(hunting): authenticate will time out at 1408.530279s 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02259c640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fd0241422b0 0x7fd02413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:54646/0 (socket says 172.21.15.138:54646) 2024-03-29T11:29:47.217 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02259c640 1 -- 172.21.15.138:0/1413911470 learned_addr learned my addr 172.21.15.138:0/1413911470 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941091011 0 0) 0x7fd0240aae40 con 0x7fd0240b17d0 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02159a640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.221+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd008003520 con 0x7fd0240b17d0 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295614133 0 0) 0x7fd02413fbd0 con 0x7fd0241422b0 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0240aae40 con 0x7fd0241422b0 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3697005090 0 0) 0x7fd008003520 con 0x7fd0240b17d0 2024-03-29T11:29:47.218 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 >> v1:172.21.15.138:6790/0 conn(0x7fd0241422b0 legacy=0x7fd02413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 >> v1:172.21.15.138:6789/0 conn(0x7fd024142690 legacy=0x7fd0240b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 monclient: found mon.noname-c 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd024151940 con 0x7fd0240b17d0 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient: _finish_auth 0 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fd01c001a10 con 0x7fd0240b17d0 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd02159a640 10 monclient: dump: 2024-03-29T11:29:47.219 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd0288ae640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.222+0000 7fd0288ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd01c001bf0 con 0x7fd0240b17d0 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd02159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd02159a640 1 -- 172.21.15.138:0/1413911470 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fd01c001f80 con 0x7fd0240b17d0 2024-03-29T11:29:47.220 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd02159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd02159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd02159a640 10 monclient: dump: 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd0288ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.223+0000 7fd0288ae640 4 set_mon_vals no callback set 2024-03-29T11:29:47.221 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.224+0000 7fd0288ae640 10 monclient: shutdown 2024-03-29T11:29:47.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.224+0000 7fd0288ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:47.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.224+0000 7fd0288ae640 1 -- 172.21.15.138:0/1413911470 >> v1:172.21.15.152:6789/0 conn(0x7fd0240b17d0 legacy=0x7fd0240b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.224+0000 7fd0288ae640 1 -- 172.21.15.138:0/1413911470 shutdown_connections 2024-03-29T11:29:47.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.224+0000 7fd0288ae640 1 -- 172.21.15.138:0/1413911470 >> 172.21.15.138:0/1413911470 conn(0x7fd024066a00 msgr2=0x7fd024066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:47.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.224+0000 7fd0288ae640 1 -- 172.21.15.138:0/1413911470 shutdown_connections 2024-03-29T11:29:47.222 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.224+0000 7fd0288ae640 1 -- 172.21.15.138:0/1413911470 wait complete. 2024-03-29T11:29:47.232 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 10 monclient: build_initial_monmap 2024-03-29T11:29:47.232 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 10 monclient: monmap: 2024-03-29T11:29:47.232 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:47.232 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 1 librados: starting msgr at 2024-03-29T11:29:47.233 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 1 librados: starting objecter 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 1 Processor -- start 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 1 -- start start 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 1 librados: setting wanted keys 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 1 librados: calling monclient init 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.226+0000 7fd0288ae640 10 monclient: init 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd0240b17d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd0241422b0 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd024142690 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240b5cf0 con 0x7fd0241422b0 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd024151150 con 0x7fd0240b17d0 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd021d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fd0241422b0 0x7fd0241dad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:36360/0 (socket says 172.21.15.138:36360) 2024-03-29T11:29:47.234 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240b7860 con 0x7fd024142690 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd021d9b640 1 -- 172.21.15.138:0/619656058 learned_addr learned my addr 172.21.15.138:0/619656058 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd0288ae640 10 monclient(hunting): authenticate will time out at 1408.536832s 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd01affd640 10 client.?.objecter ms_handle_connect 0x7fd0241422b0 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.227+0000 7fd01affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144555972 0 0) 0x7fd0240b5cf0 con 0x7fd0241422b0 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c004eb0 con 0x7fd0241422b0 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4020477665 0 0) 0x7fd00c004eb0 con 0x7fd0241422b0 2024-03-29T11:29:47.235 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0240b5cf0 con 0x7fd0241422b0 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 client.?.objecter ms_handle_connect 0x7fd024142690 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fd014001990 con 0x7fd0241422b0 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): dump: 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:47.236 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040023333 0 0) 0x7fd0240b7860 con 0x7fd024142690 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.228+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c004eb0 con 0x7fd024142690 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1772835159 0 0) 0x7fd0240b5cf0 con 0x7fd0241422b0 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 >> v1:172.21.15.138:6790/0 conn(0x7fd024142690 legacy=0x7fd0241e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 >> v1:172.21.15.152:6789/0 conn(0x7fd0240b17d0 legacy=0x7fd0240b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 1 monclient: found mon.a 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:47.237 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0241da890 con 0x7fd0241422b0 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 10 monclient: _renew_subs 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0241418d0 con 0x7fd0241422b0 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 10 monclient: _renew_subs 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd024150780 con 0x7fd0241422b0 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd0288ae640 1 librados: init done 2024-03-29T11:29:47.238 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd014001af0 con 0x7fd0241422b0 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fd014003a60 con 0x7fd0241422b0 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd01affd640 10 monclient: dump: 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:47.239 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.229+0000 7fd020d99640 4 set_mon_vals no callback set 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.231+0000 7fd0288ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.231+0000 7fd0288ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.231+0000 7fd0288ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.231+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd000005080 con 0x7fd0241422b0 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.232+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fd014003bc0 con 0x7fd0241422b0 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.233+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7fd014084b80 con 0x7fd0241422b0 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.233+0000 7fd01affd640 10 client.4554.objecter ms_dispatch 0x7fd024000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.233+0000 7fd01affd640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.233+0000 7fd01affd640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.233+0000 7fd01affd640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.233+0000 7fd02259c640 10 monclient: get_auth_request con 0x7fd00c06dc40 auth_method 0 2024-03-29T11:29:47.240 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.235+0000 7fd01affd640 10 client.4554.objecter ms_handle_connect 0x7fd00c06dc40 2024-03-29T11:29:47.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.243+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd014050460 con 0x7fd0241422b0 2024-03-29T11:29:47.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.243+0000 7fd01affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.241 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.243+0000 7fd01affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:47.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.584+0000 7fd0288ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.584+0000 7fd0288ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.584+0000 7fd0288ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:29:47.576 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.584+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd000005330 con 0x7fd0241422b0 2024-03-29T11:29:47.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.585+0000 7fd01affd640 1 -- 172.21.15.138:0/619656058 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7fd014052390 con 0x7fd0241422b0 2024-03-29T11:29:47.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.585+0000 7fd01affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.577 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.585+0000 7fd01affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:47.577 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:47.578 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:47.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 10 librados: watch_flush enter 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 10 librados: watch_flush exit 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 >> v1:172.21.15.152:6816/3052585572 conn(0x7fd00c06dc40 legacy=0x7fd00c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 10 monclient: shutdown 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 >> v1:172.21.15.138:6789/0 conn(0x7fd0241422b0 legacy=0x7fd0241dad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 shutdown_connections 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 >> 172.21.15.138:0/619656058 conn(0x7fd024066a00 msgr2=0x7fd0240b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 shutdown_connections 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 1 -- 172.21.15.138:0/619656058 wait complete. 2024-03-29T11:29:47.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:47.589+0000 7fd0288ae640 1 librados: shutdown 2024-03-29T11:29:47.598 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:54.404 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:54.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.561+0000 7f5375d2b640 10 monclient: get_monmap_and_config 2024-03-29T11:29:54.553 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.561+0000 7f5375d2b640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.561+0000 7f5375d2b640 10 monclient: monmap: 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:29:54.562038+0000 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:29:54.562038+0000 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:29:54.554 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:54.555 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.563+0000 7f5375d2b640 1 Processor -- start 2024-03-29T11:29:54.556 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.564+0000 7f5375d2b640 1 -- start start 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.564+0000 7f5375d2b640 10 monclient: init 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f53701422b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f5370142690 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f53700b17d0 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5370142c80 con 0x7f53700b17d0 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53700aae40 con 0x7f53701422b0 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537013fbd0 con 0x7f5370142690 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.557 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.565+0000 7f5375d2b640 10 monclient(hunting): authenticate will time out at 1415.873681s 2024-03-29T11:29:54.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.568+0000 7f536ffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f53700b17d0 0x7f53700b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:49590/0 (socket says 172.21.15.138:49590) 2024-03-29T11:29:54.559 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.568+0000 7f536ffff640 1 -- 172.21.15.138:0/442157070 learned_addr learned my addr 172.21.15.138:0/442157070 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:54.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185172266 0 0) 0x7f53700aae40 con 0x7f53701422b0 2024-03-29T11:29:54.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.563 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5358003520 con 0x7f53701422b0 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948469109 0 0) 0x7f5370142c80 con 0x7f53700b17d0 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.569+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53700aae40 con 0x7f53700b17d0 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2962328423 0 0) 0x7f5358003520 con 0x7f53701422b0 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 >> v1:172.21.15.138:6790/0 conn(0x7f5370142690 legacy=0x7f53700b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:54.564 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 >> v1:172.21.15.138:6789/0 conn(0x7f53700b17d0 legacy=0x7f53700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370151940 con 0x7f53701422b0 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f5375d2b640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f5375d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f53640015c0 con 0x7f53701422b0 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.570+0000 7f536e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 10 monclient: dump: 2024-03-29T11:29:54.565 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f5375d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53640017a0 con 0x7f53701422b0 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.566 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f5375d2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f5375d2b640 4 set_mon_vals no callback set 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 1 -- 172.21.15.138:0/442157070 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5364001b70 con 0x7f53701422b0 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.571+0000 7f536e7fc640 10 monclient: dump: 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:54.567 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.572+0000 7f5375d2b640 10 monclient: shutdown 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.572+0000 7f5375d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.572+0000 7f5375d2b640 1 -- 172.21.15.138:0/442157070 >> v1:172.21.15.152:6789/0 conn(0x7f53701422b0 legacy=0x7f537013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.572+0000 7f5375d2b640 1 -- 172.21.15.138:0/442157070 shutdown_connections 2024-03-29T11:29:54.568 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.572+0000 7f5375d2b640 1 -- 172.21.15.138:0/442157070 >> 172.21.15.138:0/442157070 conn(0x7f5370066a00 msgr2=0x7f5370066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.572+0000 7f5375d2b640 1 -- 172.21.15.138:0/442157070 shutdown_connections 2024-03-29T11:29:54.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.572+0000 7f5375d2b640 1 -- 172.21.15.138:0/442157070 wait complete. 2024-03-29T11:29:54.569 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.573+0000 7f5375d2b640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.573+0000 7f5375d2b640 10 monclient: monmap: 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.573+0000 7f5375d2b640 1 librados: starting msgr at 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.573+0000 7f5375d2b640 1 librados: starting objecter 2024-03-29T11:29:54.570 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.574+0000 7f5375d2b640 1 Processor -- start 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.574+0000 7f5375d2b640 1 -- start start 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.574+0000 7f5375d2b640 1 librados: setting wanted keys 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.574+0000 7f5375d2b640 1 librados: calling monclient init 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.574+0000 7f5375d2b640 10 monclient: init 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.575+0000 7f5375d2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.575+0000 7f5375d2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.575+0000 7f5375d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f53700b17d0 addr v1:172.21.15.152:6789/0 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.576+0000 7f5375d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f5370142690 addr v1:172.21.15.138:6790/0 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.576+0000 7f5375d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f537013bb80 addr v1:172.21.15.138:6789/0 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.576+0000 7f5375d2b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5370134930 con 0x7f537013bb80 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.576+0000 7f5375d2b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53700b0ec0 con 0x7f53700b17d0 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.576+0000 7f5375d2b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53700b11a0 con 0x7f5370142690 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.576+0000 7f5375d2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.576+0000 7f5375d2b640 10 monclient(hunting): authenticate will time out at 1415.884215s 2024-03-29T11:29:54.571 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.578+0000 7f536ffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f537013bb80 0x7f53700afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:52546/0 (socket says 172.21.15.138:52546) 2024-03-29T11:29:54.572 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.578+0000 7f536ffff640 1 -- 172.21.15.138:0/3350364938 learned_addr learned my addr 172.21.15.138:0/3350364938 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 client.?.objecter ms_handle_connect 0x7f5370142690 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 client.?.objecter ms_handle_connect 0x7f537013bb80 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351003241 0 0) 0x7f53700b11a0 con 0x7f5370142690 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.579+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535c004eb0 con 0x7f5370142690 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3425873831 0 0) 0x7f535c004eb0 con 0x7f5370142690 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53700b11a0 con 0x7f5370142690 2024-03-29T11:29:54.578 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f53600017d0 con 0x7f5370142690 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 10 monclient(hunting): dump: 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:54.579 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 10 client.?.objecter ms_handle_connect 0x7f53700b17d0 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749546917 0 0) 0x7f5370134930 con 0x7f537013bb80 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.581+0000 7f5357fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535c004eb0 con 0x7f537013bb80 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 473164132 0 0) 0x7f53700b11a0 con 0x7f5370142690 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 >> v1:172.21.15.152:6789/0 conn(0x7f53700b17d0 legacy=0x7f537014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 >> v1:172.21.15.138:6789/0 conn(0x7f537013bb80 legacy=0x7f53700afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 1 monclient: found mon.c 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.580 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53700af190 con 0x7f5370142690 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5375d2b640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5375d2b640 10 monclient: _renew_subs 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5375d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53700aee20 con 0x7f5370142690 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5360001930 con 0x7f5370142690 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5357fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f5375d2b640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.582+0000 7f536dffb640 4 set_mon_vals no callback set 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5375d2b640 10 monclient: _renew_subs 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5375d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:54.581 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f537014e120 con 0x7f5370142690 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5375d2b640 1 librados: init done 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5360003810 con 0x7f5370142690 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5357fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.583+0000 7f5357fff640 10 monclient: dump: 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:29:54.582 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.584+0000 7f5375d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.584+0000 7f5375d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.584+0000 7f5375d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.584+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5370075f10 con 0x7f5370142690 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.585+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f5360003970 con 0x7f5370142690 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.585+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 533924679 0 0) 0x7f5360083800 con 0x7f5370142690 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.586+0000 7f5357fff640 10 client.4622.objecter ms_dispatch 0x7f5370000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.586+0000 7f5357fff640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.586+0000 7f5357fff640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.586+0000 7f5357fff640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:29:54.583 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.586+0000 7f536f7fe640 10 monclient: get_auth_request con 0x7f535c06dd30 auth_method 0 2024-03-29T11:29:54.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.587+0000 7f5357fff640 10 client.4622.objecter ms_handle_connect 0x7f535c06dd30 2024-03-29T11:29:54.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.593+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f536004ef10 con 0x7f5370142690 2024-03-29T11:29:54.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.593+0000 7f5357fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.584 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.593+0000 7f5357fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:54.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.925+0000 7f5375d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.925+0000 7f5375d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.925+0000 7f5375d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:29:54.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.925+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53700af320 con 0x7f5370142690 2024-03-29T11:29:54.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.926+0000 7f5357fff640 1 -- 172.21.15.138:0/3350364938 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3134419876) 0x7f5360051010 con 0x7f5370142690 2024-03-29T11:29:54.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.926+0000 7f5357fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.926+0000 7f5357fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:54.918 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:29:54.918 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:19:58.092925+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:58.092925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:54.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.930+0000 7f5375d2b640 10 librados: watch_flush enter 2024-03-29T11:29:54.921 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.930+0000 7f5375d2b640 10 librados: watch_flush exit 2024-03-29T11:29:54.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.930+0000 7f5375d2b640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:29:54.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.930+0000 7f5375d2b640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:29:54.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.931+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 >> v1:172.21.15.152:6816/3052585572 conn(0x7f535c06dd30 legacy=0x7f535c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.931+0000 7f5375d2b640 10 monclient: shutdown 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.931+0000 7f5375d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.931+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 >> v1:172.21.15.138:6790/0 conn(0x7f5370142690 legacy=0x7f53700af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.931+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 shutdown_connections 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.931+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 >> 172.21.15.138:0/3350364938 conn(0x7f5370066a00 msgr2=0x7f53700b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.932+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 shutdown_connections 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.932+0000 7f5375d2b640 1 -- 172.21.15.138:0/3350364938 wait complete. 2024-03-29T11:29:54.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:29:54.932+0000 7f5375d2b640 1 librados: shutdown 2024-03-29T11:29:54.943 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:01.750 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:01.901 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.908+0000 7f5298d0b640 10 monclient: get_monmap_and_config 2024-03-29T11:30:01.901 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.908+0000 7f5298d0b640 10 monclient: build_initial_monmap 2024-03-29T11:30:01.901 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.908+0000 7f5298d0b640 10 monclient: monmap: 2024-03-29T11:30:01.901 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:01.901 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:01.909961+0000 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:01.909961+0000 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:01.902 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.909+0000 7f5298d0b640 1 Processor -- start 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.909+0000 7f5298d0b640 1 -- start start 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.909+0000 7f5298d0b640 10 monclient: init 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.910+0000 7f5298d0b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.910+0000 7f5298d0b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.910+0000 7f5298d0b640 10 monclient(hunting): picked mon.noname-c con 0x7f528c038c60 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.910+0000 7f5298d0b640 10 monclient(hunting): picked mon.noname-b con 0x7f528c0324b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.911+0000 7f5298d0b640 10 monclient(hunting): picked mon.noname-a con 0x7f528c0d11e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.911+0000 7f5298d0b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c0caf80 con 0x7f528c0d11e0 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.911+0000 7f5298d0b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c039850 con 0x7f528c038c60 2024-03-29T11:30:01.903 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.911+0000 7f5298d0b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c0c78d0 con 0x7f528c0324b0 2024-03-29T11:30:01.904 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.911+0000 7f5298d0b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:01.904 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.911+0000 7f5298d0b640 10 monclient(hunting): authenticate will time out at 1423.220110s 2024-03-29T11:30:01.908 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.915+0000 7f52937fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f528c038c60 0x7f528c0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:41245/0 (socket says 172.21.15.138:41245) 2024-03-29T11:30:01.908 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.915+0000 7f52937fe640 1 -- 172.21.15.138:0/358955464 learned_addr learned my addr 172.21.15.138:0/358955464 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:01.912 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.917+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268161899 0 0) 0x7f528c0c78d0 con 0x7f528c0324b0 2024-03-29T11:30:01.912 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.917+0000 7f52927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.917+0000 7f52927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.917+0000 7f52927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.917+0000 7f52927fc640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.917+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5284003520 con 0x7f528c0324b0 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069120450 0 0) 0x7f528c039850 con 0x7f528c038c60 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c0c78d0 con 0x7f528c038c60 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 833899971 0 0) 0x7f5284003520 con 0x7f528c0324b0 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 >> v1:172.21.15.152:6789/0 conn(0x7f528c038c60 legacy=0x7f528c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 >> v1:172.21.15.138:6789/0 conn(0x7f528c0d11e0 legacy=0x7f528c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.913 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 monclient: found mon.noname-b 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c0c7a60 con 0x7f528c0324b0 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5280001690 con 0x7f528c0324b0 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: dump: 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:01.914 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f5298d0b640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f5298d0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52800017f0 con 0x7f528c0324b0 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 1 -- 172.21.15.138:0/358955464 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5280001c00 con 0x7f528c0324b0 2024-03-29T11:30:01.915 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f52927fc640 10 monclient: dump: 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f5298d0b640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.918+0000 7f5298d0b640 4 set_mon_vals no callback set 2024-03-29T11:30:01.916 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.919+0000 7f5298d0b640 10 monclient: shutdown 2024-03-29T11:30:01.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.919+0000 7f5298d0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:01.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.919+0000 7f5298d0b640 1 -- 172.21.15.138:0/358955464 >> v1:172.21.15.138:6790/0 conn(0x7f528c0324b0 legacy=0x7f528c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.917 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.924+0000 7f5298d0b640 1 -- 172.21.15.138:0/358955464 shutdown_connections 2024-03-29T11:30:01.918 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.924+0000 7f5298d0b640 1 -- 172.21.15.138:0/358955464 >> 172.21.15.138:0/358955464 conn(0x7f528c005c20 msgr2=0x7f528c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:01.918 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.924+0000 7f5298d0b640 1 -- 172.21.15.138:0/358955464 shutdown_connections 2024-03-29T11:30:01.919 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.924+0000 7f5298d0b640 1 -- 172.21.15.138:0/358955464 wait complete. 2024-03-29T11:30:01.919 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.925+0000 7f5298d0b640 10 monclient: build_initial_monmap 2024-03-29T11:30:01.919 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.925+0000 7f5298d0b640 10 monclient: monmap: 2024-03-29T11:30:01.919 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.925+0000 7f5298d0b640 1 librados: starting msgr at 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.925+0000 7f5298d0b640 1 librados: starting objecter 2024-03-29T11:30:01.920 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.925+0000 7f5298d0b640 1 Processor -- start 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.926+0000 7f5298d0b640 1 -- start start 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.926+0000 7f5298d0b640 1 librados: setting wanted keys 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.926+0000 7f5298d0b640 1 librados: calling monclient init 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.926+0000 7f5298d0b640 10 monclient: init 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 10 monclient(hunting): picked mon.noname-c con 0x7f528c0324b0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 10 monclient(hunting): picked mon.noname-a con 0x7f528c038c60 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 10 monclient(hunting): picked mon.noname-b con 0x7f528c0d11e0 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:01.922 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c00be20 con 0x7f528c038c60 2024-03-29T11:30:01.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c16dbd0 con 0x7f528c0d11e0 2024-03-29T11:30:01.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c16dd60 con 0x7f528c0324b0 2024-03-29T11:30:01.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:01.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.927+0000 7f5298d0b640 10 monclient(hunting): authenticate will time out at 1423.236744s 2024-03-29T11:30:01.923 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.931+0000 7f5293fff640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f528c0d11e0 0x7f528c174430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:41343/0 (socket says 172.21.15.138:41343) 2024-03-29T11:30:01.925 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.931+0000 7f5293fff640 1 -- 172.21.15.138:0/1016477947 learned_addr learned my addr 172.21.15.138:0/1016477947 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.931+0000 7f527ffff640 10 client.?.objecter ms_handle_connect 0x7f528c0d11e0 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.931+0000 7f527ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 client.?.objecter ms_handle_connect 0x7f528c038c60 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982858295 0 0) 0x7f528c16dbd0 con 0x7f528c0d11e0 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5278004eb0 con 0x7f528c0d11e0 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028935198 0 0) 0x7f528c00be20 con 0x7f528c038c60 2024-03-29T11:30:01.926 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c16dbd0 con 0x7f528c038c60 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4163128210 0 0) 0x7f5278004eb0 con 0x7f528c0d11e0 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f528c00be20 con 0x7f528c0d11e0 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.932+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f529404fdd0 con 0x7f528c0d11e0 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.933+0000 7f527ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.933+0000 7f527ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.933+0000 7f527ffff640 10 monclient(hunting): dump: 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.927 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.933+0000 7f527ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.933+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3546177397 0 0) 0x7f528c00be20 con 0x7f528c0d11e0 2024-03-29T11:30:01.928 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.933+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 >> v1:172.21.15.138:6790/0 conn(0x7f528c0324b0 legacy=0x7f528c16d4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:01.932 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.935+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 >> v1:172.21.15.138:6789/0 conn(0x7f528c038c60 legacy=0x7f528c00a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.935+0000 7f527ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.935+0000 7f527ffff640 1 monclient: found mon.b 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c00ab30 con 0x7f528c0d11e0 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 10 monclient: _renew_subs 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c0097d0 con 0x7f528c0d11e0 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:01.933 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 10 monclient: _renew_subs 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c00c1d0 con 0x7f528c0d11e0 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5298d0b640 1 librados: init done 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f529404f400 con 0x7f528c0d11e0 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f527ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.936+0000 7f5291ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f5294075710 con 0x7f528c0d11e0 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f527ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f527ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f527ffff640 10 monclient: dump: 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.934 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f5298d0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f5298d0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f5298d0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.937+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5274005080 con 0x7f528c0d11e0 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.938+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f5294067bf0 con 0x7f528c0d11e0 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.939+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7f52940ff5d0 con 0x7f528c0d11e0 2024-03-29T11:30:01.935 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.939+0000 7f527ffff640 10 client.4522.objecter ms_dispatch 0x7f5294000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:01.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.939+0000 7f527ffff640 3 client.4522.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:01.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.939+0000 7f527ffff640 3 client.4522.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:01.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.939+0000 7f527ffff640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:30:01.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.942+0000 7f52937fe640 10 monclient: get_auth_request con 0x7f527806db10 auth_method 0 2024-03-29T11:30:01.936 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.943+0000 7f527ffff640 10 client.4522.objecter ms_handle_connect 0x7f527806db10 2024-03-29T11:30:01.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.946+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52940caeb0 con 0x7f528c0d11e0 2024-03-29T11:30:01.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.946+0000 7f527ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.939 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:01.946+0000 7f527ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:02.270 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.278+0000 7f5298d0b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.278+0000 7f5298d0b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.278+0000 7f5298d0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:02.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.278+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5274005330 con 0x7f528c0d11e0 2024-03-29T11:30:02.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.279+0000 7f527ffff640 1 -- 172.21.15.138:0/1016477947 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7f52940ccde0 con 0x7f528c0d11e0 2024-03-29T11:30:02.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.279+0000 7f527ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.279+0000 7f527ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:02.272 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:02.272 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:02.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.282+0000 7f5298d0b640 10 librados: watch_flush enter 2024-03-29T11:30:02.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.282+0000 7f5298d0b640 10 librados: watch_flush exit 2024-03-29T11:30:02.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.283+0000 7f5298d0b640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:30:02.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.283+0000 7f5298d0b640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:30:02.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.283+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 >> v1:172.21.15.152:6816/3052585572 conn(0x7f527806db10 legacy=0x7f527806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.283+0000 7f5298d0b640 10 monclient: shutdown 2024-03-29T11:30:02.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.283+0000 7f5298d0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:02.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.283+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 >> v1:172.21.15.152:6789/0 conn(0x7f528c0d11e0 legacy=0x7f528c174430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.284+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 shutdown_connections 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.284+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 >> 172.21.15.138:0/1016477947 conn(0x7f528c005c20 msgr2=0x7f528c0c5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.285+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 shutdown_connections 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.285+0000 7f5298d0b640 1 -- 172.21.15.138:0/1016477947 wait complete. 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:02.285+0000 7f5298d0b640 1 librados: shutdown 2024-03-29T11:30:02.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:09.102 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.259+0000 7fdb6c463640 10 monclient: get_monmap_and_config 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.259+0000 7fdb6c463640 10 monclient: build_initial_monmap 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.259+0000 7fdb6c463640 10 monclient: monmap: 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:09.260356+0000 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:09.260356+0000 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:09.257 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.260+0000 7fdb6c463640 1 Processor -- start 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.260+0000 7fdb6c463640 1 -- start start 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.260+0000 7fdb6c463640 10 monclient: init 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 10 monclient(hunting): picked mon.noname-b con 0x7fdb640ad6c0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 10 monclient(hunting): picked mon.noname-c con 0x7fdb640af6c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6414fdc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640a94e0 con 0x7fdb6414fdc0 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640af3a0 con 0x7fdb640af6c0 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640afb60 con 0x7fdb640ad6c0 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:09.258 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.261+0000 7fdb6c463640 10 monclient(hunting): authenticate will time out at 1430.570862s 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb6a1ff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fdb640ad6c0 0x7fdb6414f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:46896/0 (socket says 172.21.15.138:46896) 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb6a1ff640 1 -- 172.21.15.138:0/2674579111 learned_addr learned my addr 172.21.15.138:0/2674579111 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936665012 0 0) 0x7fdb640afb60 con 0x7fdb640ad6c0 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb691fd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb691fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb691fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb691fd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.262+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c003500 con 0x7fdb640ad6c0 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.263+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906060954 0 0) 0x7fdb4c003500 con 0x7fdb640ad6c0 2024-03-29T11:30:09.259 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.263+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 >> v1:172.21.15.152:6789/0 conn(0x7fdb640af6c0 legacy=0x7fdb64152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:09.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 >> v1:172.21.15.138:6789/0 conn(0x7fdb6414fdc0 legacy=0x7fdb6415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:09.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:09.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 1 monclient: found mon.noname-b 2024-03-29T11:30:09.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:09.260 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb640aab80 con 0x7fdb640ad6c0 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: _finish_auth 0 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb6c463640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb6c463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb54001690 con 0x7fdb640ad6c0 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: dump: 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.261 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb6c463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb540019a0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb691fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb6c463640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.267+0000 7fdb6c463640 4 set_mon_vals no callback set 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb691fd640 1 -- 172.21.15.138:0/2674579111 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb54001d30 con 0x7fdb640ad6c0 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb691fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.262 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb691fd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb691fd640 10 monclient: dump: 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:09.263 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:09.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb6c463640 10 monclient: shutdown 2024-03-29T11:30:09.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb6c463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:09.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb6c463640 1 -- 172.21.15.138:0/2674579111 >> v1:172.21.15.138:6790/0 conn(0x7fdb640ad6c0 legacy=0x7fdb6414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb6c463640 1 -- 172.21.15.138:0/2674579111 shutdown_connections 2024-03-29T11:30:09.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb6c463640 1 -- 172.21.15.138:0/2674579111 >> 172.21.15.138:0/2674579111 conn(0x7fdb640b35e0 msgr2=0x7fdb640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:09.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.268+0000 7fdb6c463640 1 -- 172.21.15.138:0/2674579111 shutdown_connections 2024-03-29T11:30:09.264 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.269+0000 7fdb6c463640 1 -- 172.21.15.138:0/2674579111 wait complete. 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 10 monclient: build_initial_monmap 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 10 monclient: monmap: 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:09.265 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 1 librados: starting msgr at 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 1 librados: starting objecter 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 1 Processor -- start 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 1 -- start start 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 1 librados: setting wanted keys 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 1 librados: calling monclient init 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.270+0000 7fdb6c463640 10 monclient: init 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 10 monclient(hunting): picked mon.noname-c con 0x7fdb640ad6c0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 10 monclient(hunting): picked mon.noname-b con 0x7fdb640af6c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6414fdc0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6406fff0 con 0x7fdb6414fdc0 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640702d0 con 0x7fdb640af6c0 2024-03-29T11:30:09.266 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640705b0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:09.267 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.271+0000 7fdb6c463640 10 monclient(hunting): authenticate will time out at 1430.580584s 2024-03-29T11:30:09.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.274+0000 7fdb6a1ff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7fdb640ad6c0 0x7fdb640709d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:59428/0 (socket says 172.21.15.138:59428) 2024-03-29T11:30:09.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.274+0000 7fdb6a1ff640 1 -- 172.21.15.138:0/3456297167 learned_addr learned my addr 172.21.15.138:0/3456297167 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:09.271 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.274+0000 7fdb62ffd640 10 client.?.objecter ms_handle_connect 0x7fdb640ad6c0 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 client.?.objecter ms_handle_connect 0x7fdb6414fdc0 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368286949 0 0) 0x7fdb640705b0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb50004eb0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942457594 0 0) 0x7fdb6406fff0 con 0x7fdb6414fdc0 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.272 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb640705b0 con 0x7fdb6414fdc0 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.275+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000810656 0 0) 0x7fdb50004eb0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb6406fff0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb54001650 con 0x7fdb640ad6c0 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:09.273 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 client.?.objecter ms_handle_connect 0x7fdb640af6c0 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 767690675 0 0) 0x7fdb640705b0 con 0x7fdb6414fdc0 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb50004eb0 con 0x7fdb6414fdc0 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 873649614 0 0) 0x7fdb6406fff0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 >> v1:172.21.15.152:6789/0 conn(0x7fdb640af6c0 legacy=0x7fdb6414e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 >> v1:172.21.15.138:6789/0 conn(0x7fdb6414fdc0 legacy=0x7fdb641ff7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.274 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 monclient: found mon.c 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6406e7a0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 10 monclient: _renew_subs 2024-03-29T11:30:09.275 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:30:09.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6406e990 con 0x7fdb640ad6c0 2024-03-29T11:30:09.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:09.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 10 monclient: _renew_subs 2024-03-29T11:30:09.276 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6408a960 con 0x7fdb640ad6c0 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.276+0000 7fdb6c463640 1 librados: init done 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.278+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb54004c70 con 0x7fdb640ad6c0 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.278+0000 7fdb62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.278+0000 7fdb63fff640 4 set_mon_vals no callback set 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.280+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fdb54004f80 con 0x7fdb640ad6c0 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.280+0000 7fdb62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.278 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.280+0000 7fdb62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.280+0000 7fdb62ffd640 10 monclient: dump: 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.280+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fdb5401af50 con 0x7fdb640ad6c0 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb6c463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.279 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb6c463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb6c463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb44005080 con 0x7fdb640ad6c0 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7fdb54084c60 con 0x7fdb640ad6c0 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb62ffd640 10 client.4631.objecter ms_dispatch 0x7fdb64000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb62ffd640 3 client.4631.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb62ffd640 3 client.4631.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb62ffd640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb699fe640 10 monclient: get_auth_request con 0x7fdb5006ddd0 auth_method 0 2024-03-29T11:30:09.280 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.281+0000 7fdb62ffd640 10 client.4631.objecter ms_handle_connect 0x7fdb5006ddd0 2024-03-29T11:30:09.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.290+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb540505c0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.290+0000 7fdb62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.283 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.290+0000 7fdb62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:09.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.617+0000 7fdb6c463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.617+0000 7fdb6c463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.617+0000 7fdb6c463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.138:6790/0 2024-03-29T11:30:09.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.617+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 --> v1:172.21.15.138:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb44005330 con 0x7fdb640ad6c0 2024-03-29T11:30:09.611 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.618+0000 7fdb62ffd640 1 -- 172.21.15.138:0/3456297167 <== mon.2 v1:172.21.15.138:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7fdb540017b0 con 0x7fdb640ad6c0 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.618+0000 7fdb62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.618+0000 7fdb62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:09.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 10 librados: watch_flush enter 2024-03-29T11:30:09.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 10 librados: watch_flush exit 2024-03-29T11:30:09.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 >> v1:172.21.15.152:6816/3052585572 conn(0x7fdb5006ddd0 legacy=0x7fdb50070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 10 monclient: shutdown 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 >> v1:172.21.15.138:6790/0 conn(0x7fdb640ad6c0 legacy=0x7fdb640709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 shutdown_connections 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 >> 172.21.15.138:0/3456297167 conn(0x7fdb640b35e0 msgr2=0x7fdb64152980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 shutdown_connections 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.622+0000 7fdb6c463640 1 -- 172.21.15.138:0/3456297167 wait complete. 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:09.623+0000 7fdb6c463640 1 librados: shutdown 2024-03-29T11:30:09.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:16.442 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.597+0000 7f9348c20640 10 monclient: get_monmap_and_config 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.597+0000 7f9348c20640 10 monclient: build_initial_monmap 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.597+0000 7f9348c20640 10 monclient: monmap: 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:16.598576+0000 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:16.598576+0000 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:16.591 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.598+0000 7f9348c20640 1 Processor -- start 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.599+0000 7f9348c20640 1 -- start start 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.599+0000 7f9348c20640 10 monclient: init 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 10 monclient(hunting): picked mon.noname-c con 0x7f93440a9ac0 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 10 monclient(hunting): picked mon.noname-a con 0x7f934413ee90 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 10 monclient(hunting): picked mon.noname-b con 0x7f93440af900 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93440afd10 con 0x7f934413ee90 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93440af6f0 con 0x7f93440a9ac0 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93440aac80 con 0x7f93440af900 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:16.593 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.600+0000 7f9348c20640 10 monclient(hunting): authenticate will time out at 1437.909748s 2024-03-29T11:30:16.596 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.603+0000 7f9342d9d640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f93440af900 0x7f9344149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:48124/0 (socket says 172.21.15.138:48124) 2024-03-29T11:30:16.596 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.603+0000 7f9342d9d640 1 -- 172.21.15.138:0/1929259432 learned_addr learned my addr 172.21.15.138:0/1929259432 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:16.596 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.603+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17366973 0 0) 0x7f93440aac80 con 0x7f93440af900 2024-03-29T11:30:16.596 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.596 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:30:16.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f932c003500 con 0x7f93440af900 2024-03-29T11:30:16.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217295973 0 0) 0x7f932c003500 con 0x7f93440af900 2024-03-29T11:30:16.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 >> v1:172.21.15.152:6789/0 conn(0x7f93440a9ac0 legacy=0x7f93440adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:16.597 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 >> v1:172.21.15.138:6789/0 conn(0x7f934413ee90 legacy=0x7f934413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient: _finish_hunting 0 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 1 monclient: found mon.noname-b 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.138:6790/0 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 --> v1:172.21.15.138:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93440aaf60 con 0x7f93440af900 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient: _finish_auth 0 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient: _check_auth_tickets 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f933c005030 con 0x7f93440af900 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f934159a640 10 monclient: dump: 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:16.598 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f9348c20640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:30:16.599 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.604+0000 7f9348c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:16.600 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 <== mon.2 v1:172.21.15.138:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f933c005200 con 0x7f93440af900 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f934159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f934159a640 1 -- 172.21.15.138:0/1929259432 <== mon.2 v1:172.21.15.138:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f933c001770 con 0x7f93440af900 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f934159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f934159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f934159a640 10 monclient: dump: 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:16.601 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f9348c20640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f9348c20640 4 set_mon_vals no callback set 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f9348c20640 10 monclient: shutdown 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f9348c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.606+0000 7f9348c20640 1 -- 172.21.15.138:0/1929259432 >> v1:172.21.15.138:6790/0 conn(0x7f93440af900 legacy=0x7f9344149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.607+0000 7f9348c20640 1 -- 172.21.15.138:0/1929259432 shutdown_connections 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.607+0000 7f9348c20640 1 -- 172.21.15.138:0/1929259432 >> 172.21.15.138:0/1929259432 conn(0x7f93440b35e0 msgr2=0x7f93440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.608+0000 7f9348c20640 1 -- 172.21.15.138:0/1929259432 shutdown_connections 2024-03-29T11:30:16.602 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.608+0000 7f9348c20640 1 -- 172.21.15.138:0/1929259432 wait complete. 2024-03-29T11:30:16.603 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.609+0000 7f9348c20640 10 monclient: build_initial_monmap 2024-03-29T11:30:16.603 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.609+0000 7f9348c20640 10 monclient: monmap: 2024-03-29T11:30:16.603 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:16.603 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:16.603 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:16.603 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:16.603 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.609+0000 7f9348c20640 1 librados: starting msgr at 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.609+0000 7f9348c20640 1 librados: starting objecter 2024-03-29T11:30:16.604 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.609+0000 7f9348c20640 1 Processor -- start 2024-03-29T11:30:16.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 1 -- start start 2024-03-29T11:30:16.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 1 librados: setting wanted keys 2024-03-29T11:30:16.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 1 librados: calling monclient init 2024-03-29T11:30:16.605 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 10 monclient: init 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 10 monclient(hunting): picked mon.noname-a con 0x7f93440a9ac0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.610+0000 7f9348c20640 10 monclient(hunting): picked mon.noname-c con 0x7f93440af900 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.611+0000 7f9341d9b640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f93440af900 0x7f93441383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:49546/0 (socket says 172.21.15.138:49546) 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.612+0000 7f9348c20640 10 monclient(hunting): picked mon.noname-b con 0x7f934413ee90 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.612+0000 7f9348c20640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93441dbec0 con 0x7f93440a9ac0 2024-03-29T11:30:16.615 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.612+0000 7f9348c20640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9344070c30 con 0x7f934413ee90 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.612+0000 7f9348c20640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93441425d0 con 0x7f93440af900 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.612+0000 7f9348c20640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.612+0000 7f9348c20640 10 monclient(hunting): authenticate will time out at 1437.921174s 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9341d9b640 1 -- 172.21.15.138:0/912529244 learned_addr learned my addr 172.21.15.138:0/912529244 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 client.?.objecter ms_handle_connect 0x7f93440af900 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 client.?.objecter ms_handle_connect 0x7f93440a9ac0 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279756448 0 0) 0x7f93441dbec0 con 0x7f93440a9ac0 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:30:16.616 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9324004eb0 con 0x7f93440a9ac0 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 client.?.objecter ms_handle_connect 0x7f934413ee90 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762357681 0 0) 0x7f93441425d0 con 0x7f93440af900 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93441dbec0 con 0x7f93440af900 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040012387 0 0) 0x7f9344070c30 con 0x7f934413ee90 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:30:16.617 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.616+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9324005360 con 0x7f934413ee90 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3564791548 0 0) 0x7f9324004eb0 con 0x7f93440a9ac0 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9344070c30 con 0x7f93440a9ac0 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9334001510 con 0x7f93440a9ac0 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:16.618 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3824237171 0 0) 0x7f93441dbec0 con 0x7f93440af900 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9324004eb0 con 0x7f93440af900 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f93380018d0 con 0x7f93440af900 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.619 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2833690699 0 0) 0x7f9344070c30 con 0x7f93440a9ac0 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 >> v1:172.21.15.138:6790/0 conn(0x7f93440af900 legacy=0x7f93441383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 >> v1:172.21.15.152:6789/0 conn(0x7f934413ee90 legacy=0x7f93441e74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 monclient: found mon.a 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93441dca60 con 0x7f93440a9ac0 2024-03-29T11:30:16.620 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 10 monclient: _renew_subs 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934406f130 con 0x7f93440a9ac0 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 10 monclient: _renew_subs 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f934406ea10 con 0x7f93440a9ac0 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.617+0000 7f9348c20640 1 librados: init done 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.619+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9334001aa0 con 0x7f93440a9ac0 2024-03-29T11:30:16.621 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.619+0000 7f9332ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.619+0000 7f9340d99640 4 set_mon_vals no callback set 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9348c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f9334001d50 con 0x7f93440a9ac0 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9348c20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9348c20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9328005080 con 0x7f93440a9ac0 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9332ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9332ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.620+0000 7f9332ffd640 10 monclient: dump: 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.622 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.621+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f9334019920 con 0x7f93440a9ac0 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.621+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7f9334083720 con 0x7f93440a9ac0 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.621+0000 7f9332ffd640 10 client.4575.objecter ms_dispatch 0x7f9344000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.621+0000 7f9332ffd640 3 client.4575.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.621+0000 7f9332ffd640 3 client.4575.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.621+0000 7f9332ffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.622+0000 7f9341d9b640 10 monclient: get_auth_request con 0x7f932406e0b0 auth_method 0 2024-03-29T11:30:16.623 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.622+0000 7f9332ffd640 10 client.4575.objecter ms_handle_connect 0x7f932406e0b0 2024-03-29T11:30:16.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.629+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f933404ef70 con 0x7f93440a9ac0 2024-03-29T11:30:16.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.629+0000 7f9332ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.624 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.629+0000 7f9332ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:16.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.957+0000 7f9348c20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:16.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.957+0000 7f9348c20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:16.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.957+0000 7f9348c20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:16.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.957+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9328005330 con 0x7f93440a9ac0 2024-03-29T11:30:16.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.959+0000 7f9332ffd640 1 -- 172.21.15.138:0/912529244 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7f9334051070 con 0x7f93440a9ac0 2024-03-29T11:30:16.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.959+0000 7f9332ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:16.952 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.959+0000 7f9332ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:16.952 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:16.953 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.962+0000 7f9348c20640 10 librados: watch_flush enter 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 10 librados: watch_flush exit 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 10 client.4575.objecter successfully canceled tick 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 >> v1:172.21.15.152:6816/3052585572 conn(0x7f932406e0b0 legacy=0x7f9324070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 10 monclient: shutdown 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 >> v1:172.21.15.138:6789/0 conn(0x7f93440a9ac0 legacy=0x7f93441dc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 shutdown_connections 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 >> 172.21.15.138:0/912529244 conn(0x7f93440b35e0 msgr2=0x7f934413b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 shutdown_connections 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 1 -- 172.21.15.138:0/912529244 wait complete. 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:16.963+0000 7f9348c20640 1 librados: shutdown 2024-03-29T11:30:16.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:23.782 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.949+0000 7ff29aee2640 10 monclient: get_monmap_and_config 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.949+0000 7ff29aee2640 10 monclient: build_initial_monmap 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.950+0000 7ff29aee2640 10 monclient: monmap: 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:23.951274+0000 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:23.951274+0000 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:23.943 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:23.944 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:23.944 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:23.944 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:23.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.951+0000 7ff29aee2640 1 Processor -- start 2024-03-29T11:30:23.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.951+0000 7ff29aee2640 1 -- start start 2024-03-29T11:30:23.944 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.951+0000 7ff29aee2640 10 monclient: init 2024-03-29T11:30:23.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:23.945 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 10 monclient(hunting): picked mon.noname-a con 0x7ff2940b0770 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 10 monclient(hunting): picked mon.noname-b con 0x7ff2940ad850 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 10 monclient(hunting): picked mon.noname-c con 0x7ff29413c1b0 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2940ac8e0 con 0x7ff2940b0770 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2940ad330 con 0x7ff29413c1b0 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2940acfe0 con 0x7ff2940ad850 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.952+0000 7ff29aee2640 10 monclient(hunting): authenticate will time out at 1445.261974s 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff29947f640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ff29413c1b0 0x7ff29414b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:42850/0 (socket says 172.21.15.138:42850) 2024-03-29T11:30:23.946 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff29947f640 1 -- 172.21.15.138:0/1638690179 learned_addr learned my addr 172.21.15.138:0/1638690179 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896359716 0 0) 0x7ff2940ad330 con 0x7ff29413c1b0 2024-03-29T11:30:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:30:23.947 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff284003520 con 0x7ff29413c1b0 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654119249 0 0) 0x7ff2940acfe0 con 0x7ff2940ad850 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.953+0000 7ff2937fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.954+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff284003840 con 0x7ff2940ad850 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.954+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2303568905 0 0) 0x7ff284003520 con 0x7ff29413c1b0 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.954+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 >> v1:172.21.15.138:6790/0 conn(0x7ff2940ad850 legacy=0x7ff2940aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:23.948 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.954+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 >> v1:172.21.15.138:6789/0 conn(0x7ff2940b0770 legacy=0x7ff294072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:23.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:23.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 1 monclient: found mon.noname-c 2024-03-29T11:30:23.949 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940ab1f0 con 0x7ff29413c1b0 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff28c001920 con 0x7ff29413c1b0 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff2937fe640 10 monclient: dump: 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.950 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.951 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:23.951 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:23.951 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:23.951 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:23.951 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:23.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff29aee2640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:30:23.951 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.956+0000 7ff29aee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:23.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff28c001b00 con 0x7ff29413c1b0 2024-03-29T11:30:23.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff2937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:23.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff2937fe640 1 -- 172.21.15.138:0/1638690179 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff28c001f10 con 0x7ff29413c1b0 2024-03-29T11:30:23.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff2937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff2937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:23.953 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff2937fe640 10 monclient: dump: 2024-03-29T11:30:23.953 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff29aee2640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.957+0000 7ff29aee2640 4 set_mon_vals no callback set 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.958+0000 7ff29aee2640 10 monclient: shutdown 2024-03-29T11:30:23.954 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.958+0000 7ff29aee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.958+0000 7ff29aee2640 1 -- 172.21.15.138:0/1638690179 >> v1:172.21.15.152:6789/0 conn(0x7ff29413c1b0 legacy=0x7ff29414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.958+0000 7ff29aee2640 1 -- 172.21.15.138:0/1638690179 shutdown_connections 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.958+0000 7ff29aee2640 1 -- 172.21.15.138:0/1638690179 >> 172.21.15.138:0/1638690179 conn(0x7ff29406b660 msgr2=0x7ff29406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.958+0000 7ff29aee2640 1 -- 172.21.15.138:0/1638690179 shutdown_connections 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.958+0000 7ff29aee2640 1 -- 172.21.15.138:0/1638690179 wait complete. 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.959+0000 7ff29aee2640 10 monclient: build_initial_monmap 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.959+0000 7ff29aee2640 10 monclient: monmap: 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:23.955 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.960+0000 7ff29aee2640 1 librados: starting msgr at 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.960+0000 7ff29aee2640 1 librados: starting objecter 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.960+0000 7ff29aee2640 1 Processor -- start 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.960+0000 7ff29aee2640 1 -- start start 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.960+0000 7ff29aee2640 1 librados: setting wanted keys 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.960+0000 7ff29aee2640 1 librados: calling monclient init 2024-03-29T11:30:23.956 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.960+0000 7ff29aee2640 10 monclient: init 2024-03-29T11:30:23.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.961+0000 7ff29aee2640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:23.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.961+0000 7ff29aee2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:23.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.961+0000 7ff29aee2640 10 monclient(hunting): picked mon.noname-b con 0x7ff2940ad850 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:23.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.962+0000 7ff29aee2640 10 monclient(hunting): picked mon.noname-c con 0x7ff2940b0770 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:23.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.962+0000 7ff298c7e640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ff2940ad850 0x7ff294132880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:47464/0 (socket says 172.21.15.138:47464) 2024-03-29T11:30:23.957 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.963+0000 7ff293fff640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7ff2940b0770 0x7ff2941d84b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:43372/0 (socket says 172.21.15.138:43372) 2024-03-29T11:30:23.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.964+0000 7ff29aee2640 10 monclient(hunting): picked mon.noname-a con 0x7ff29413c1b0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:23.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.964+0000 7ff29aee2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29406e3a0 con 0x7ff29413c1b0 2024-03-29T11:30:23.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.964+0000 7ff29aee2640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29413d7c0 con 0x7ff2940ad850 2024-03-29T11:30:23.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.964+0000 7ff29aee2640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29413f7c0 con 0x7ff2940b0770 2024-03-29T11:30:23.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.964+0000 7ff29aee2640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:23.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.964+0000 7ff29aee2640 10 monclient(hunting): authenticate will time out at 1445.273273s 2024-03-29T11:30:23.964 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.964+0000 7ff298c7e640 1 -- 172.21.15.138:0/364902637 learned_addr learned my addr 172.21.15.138:0/364902637 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 client.?.objecter ms_handle_connect 0x7ff2940ad850 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 client.?.objecter ms_handle_connect 0x7ff2940b0770 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455193346 0 0) 0x7ff29413d7c0 con 0x7ff2940ad850 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff278004eb0 con 0x7ff2940ad850 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707002611 0 0) 0x7ff29413f7c0 con 0x7ff2940b0770 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:23.965 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.965+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29413d7c0 con 0x7ff2940b0770 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2769270155 0 0) 0x7ff278004eb0 con 0x7ff2940ad850 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff29413f7c0 con 0x7ff2940ad850 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 client.?.objecter ms_handle_connect 0x7ff29413c1b0 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff2800019d0 con 0x7ff2940ad850 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 monclient(hunting): dump: 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.966 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1262690643 0 0) 0x7ff29413d7c0 con 0x7ff2940b0770 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff278004eb0 con 0x7ff2940b0770 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff2880018b0 con 0x7ff2940b0770 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.966+0000 7ff2917fa640 10 monclient(hunting): dump: 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:23.967 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2345220597 0 0) 0x7ff29413f7c0 con 0x7ff2940ad850 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 >> v1:172.21.15.138:6790/0 conn(0x7ff2940b0770 legacy=0x7ff2941d84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 >> v1:172.21.15.138:6789/0 conn(0x7ff29413c1b0 legacy=0x7ff2941e0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 1 monclient: found mon.b 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:23.968 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29413f950 con 0x7ff2940ad850 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff2917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff29aee2640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff29aee2640 10 monclient: _renew_subs 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff29aee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.967+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff294132f90 con 0x7ff2940ad850 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff29aee2640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff29aee2640 10 monclient: _renew_subs 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff29aee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29413d3a0 con 0x7ff2940ad850 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff29aee2640 1 librados: init done 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff280001b30 con 0x7ff2940ad850 2024-03-29T11:30:23.969 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff2917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:23.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff280003a60 con 0x7ff2940ad850 2024-03-29T11:30:23.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff2917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff2917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:23.970 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.968+0000 7ff2917fa640 10 monclient: dump: 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:23.971 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.969+0000 7ff292ffd640 4 set_mon_vals no callback set 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.970+0000 7ff29aee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.970+0000 7ff29aee2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.970+0000 7ff29aee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.970+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff274005080 con 0x7ff2940ad850 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.970+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff280003bc0 con 0x7ff2940ad850 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.971+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7ff280084a40 con 0x7ff2940ad850 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.971+0000 7ff2917fa640 10 client.4537.objecter ms_dispatch 0x7ff294000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.971+0000 7ff2917fa640 3 client.4537.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.971+0000 7ff2917fa640 3 client.4537.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:23.972 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.971+0000 7ff2917fa640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-29T11:30:23.973 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.971+0000 7ff293fff640 10 monclient: get_auth_request con 0x7ff27806df30 auth_method 0 2024-03-29T11:30:23.973 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.972+0000 7ff2917fa640 10 client.4537.objecter ms_handle_connect 0x7ff27806df30 2024-03-29T11:30:23.973 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.978+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff280050320 con 0x7ff2940ad850 2024-03-29T11:30:23.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.978+0000 7ff2917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:23.974 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:23.978+0000 7ff2917fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:24.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.330+0000 7ff29aee2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.330+0000 7ff29aee2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.330+0000 7ff29aee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.152:6789/0 2024-03-29T11:30:24.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.330+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff274005330 con 0x7ff2940ad850 2024-03-29T11:30:24.326 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:24.326 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:24.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.331+0000 7ff2917fa640 1 -- 172.21.15.138:0/364902637 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7ff280052250 con 0x7ff2940ad850 2024-03-29T11:30:24.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.331+0000 7ff2917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.331+0000 7ff2917fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:24.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.336+0000 7ff29aee2640 10 librados: watch_flush enter 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.337+0000 7ff29aee2640 10 librados: watch_flush exit 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.337+0000 7ff29aee2640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.337+0000 7ff29aee2640 10 client.4537.objecter successfully canceled tick 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.338+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff27806df30 legacy=0x7ff2780703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.338+0000 7ff29aee2640 10 monclient: shutdown 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.338+0000 7ff29aee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.338+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 >> v1:172.21.15.152:6789/0 conn(0x7ff2940ad850 legacy=0x7ff294132880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.338+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 shutdown_connections 2024-03-29T11:30:24.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.338+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 >> 172.21.15.138:0/364902637 conn(0x7ff29406b660 msgr2=0x7ff2940ae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:24.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.339+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 shutdown_connections 2024-03-29T11:30:24.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.339+0000 7ff29aee2640 1 -- 172.21.15.138:0/364902637 wait complete. 2024-03-29T11:30:24.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:24.339+0000 7ff29aee2640 1 librados: shutdown 2024-03-29T11:30:24.351 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:31.158 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:31.310 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.316+0000 7f725564e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:31.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.316+0000 7f725564e640 10 monclient: build_initial_monmap 2024-03-29T11:30:31.311 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.317+0000 7f725564e640 10 monclient: monmap: 2024-03-29T11:30:31.311 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:31.311 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:31.311 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:31.318283+0000 2024-03-29T11:30:31.311 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:31.318283+0000 2024-03-29T11:30:31.311 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:31.312 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:31.312 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:31.312 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:31.312 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:31.312 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.317+0000 7f725564e640 1 Processor -- start 2024-03-29T11:30:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.318+0000 7f725564e640 1 -- start start 2024-03-29T11:30:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.318+0000 7f725564e640 10 monclient: init 2024-03-29T11:30:31.314 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.318+0000 7f725564e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.318+0000 7f725564e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.318+0000 7f725564e640 10 monclient(hunting): picked mon.noname-a con 0x7f725006ab70 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.318+0000 7f725564e640 10 monclient(hunting): picked mon.noname-b con 0x7f725006af50 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f725564e640 10 monclient(hunting): picked mon.noname-c con 0x7f7250141fb0 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f725564e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72500b0d50 con 0x7f725006ab70 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f725564e640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72500b1030 con 0x7f7250141fb0 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f725564e640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72500afa70 con 0x7f725006af50 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f725564e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f725564e640 10 monclient(hunting): authenticate will time out at 1452.628095s 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f724e7fc640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f725006af50 0x7f7250072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:43502/0 (socket says 172.21.15.138:43502) 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.319+0000 7f724e7fc640 1 -- 172.21.15.138:0/2001902258 learned_addr learned my addr 172.21.15.138:0/2001902258 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106645816 0 0) 0x7f72500afa70 con 0x7f725006af50 2024-03-29T11:30:31.315 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7238003520 con 0x7f725006af50 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203881412 0 0) 0x7f72500b0d50 con 0x7f725006ab70 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72500afa70 con 0x7f725006ab70 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3361441817 0 0) 0x7f72500afa70 con 0x7f725006ab70 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 >> v1:172.21.15.138:6790/0 conn(0x7f725006af50 legacy=0x7f7250072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 >> v1:172.21.15.152:6789/0 conn(0x7f7250141fb0 legacy=0x7f72500715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:31.316 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 monclient: found mon.noname-a 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72501423e0 con 0x7f725006ab70 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f72480019f0 con 0x7f725006ab70 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.320+0000 7f724dffb640 10 monclient: dump: 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.317 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f725564e640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f725564e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7248001b50 con 0x7f725006ab70 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f724dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f724dffb640 1 -- 172.21.15.138:0/2001902258 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7248001fa0 con 0x7f725006ab70 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f725564e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f725564e640 4 set_mon_vals no callback set 2024-03-29T11:30:31.318 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f724dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f724dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.321+0000 7f724dffb640 10 monclient: dump: 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:31.319 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:31.320 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:31.320 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:31.320 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:31.323 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.322+0000 7f725564e640 10 monclient: shutdown 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.322+0000 7f725564e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.322+0000 7f725564e640 1 -- 172.21.15.138:0/2001902258 >> v1:172.21.15.138:6789/0 conn(0x7f725006ab70 legacy=0x7f72500af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.322+0000 7f725564e640 1 -- 172.21.15.138:0/2001902258 shutdown_connections 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.322+0000 7f725564e640 1 -- 172.21.15.138:0/2001902258 >> 172.21.15.138:0/2001902258 conn(0x7f7250067470 msgr2=0x7f72500b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.322+0000 7f725564e640 1 -- 172.21.15.138:0/2001902258 shutdown_connections 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.322+0000 7f725564e640 1 -- 172.21.15.138:0/2001902258 wait complete. 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.323+0000 7f725564e640 10 monclient: build_initial_monmap 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.323+0000 7f725564e640 10 monclient: monmap: 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:31.324 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.323+0000 7f725564e640 1 librados: starting msgr at 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.323+0000 7f725564e640 1 librados: starting objecter 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 1 Processor -- start 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 1 -- start start 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 1 librados: setting wanted keys 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 1 librados: calling monclient init 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 10 monclient: init 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:31.325 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.324+0000 7f725564e640 10 monclient(hunting): picked mon.noname-c con 0x7f725006ab70 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f725564e640 10 monclient(hunting): picked mon.noname-a con 0x7f7250141fb0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f725564e640 10 monclient(hunting): picked mon.noname-b con 0x7f7250070760 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f724effd640 1 --1- >> v1:172.21.15.138:6790/0 conn(0x7f725006ab70 0x7f7250134430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6790/0 says I am v1:172.21.15.138:43880/0 (socket says 172.21.15.138:43880) 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f724effd640 1 -- 172.21.15.138:0/2941660212 learned_addr learned my addr 172.21.15.138:0/2941660212 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7250136f10 con 0x7f7250141fb0 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7250134b40 con 0x7f7250070760 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 client.?.objecter ms_handle_connect 0x7f725006ab70 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 client.?.objecter ms_handle_connect 0x7f7250141fb0 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72501db540 con 0x7f725006ab70 2024-03-29T11:30:31.326 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f725564e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f725564e640 10 monclient(hunting): authenticate will time out at 1452.634613s 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 client.?.objecter ms_handle_connect 0x7f7250070760 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317258659 0 0) 0x7f7250136f10 con 0x7f7250141fb0 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f723c004eb0 con 0x7f7250141fb0 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834186514 0 0) 0x7f7250134b40 con 0x7f7250070760 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:30:31.327 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f723c006230 con 0x7f7250070760 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800082078 0 0) 0x7f72501db540 con 0x7f725006ab70 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.325+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7250134b40 con 0x7f725006ab70 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 496774756 0 0) 0x7f723c004eb0 con 0x7f7250141fb0 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72501db540 con 0x7f7250141fb0 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7244001950 con 0x7f7250141fb0 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient(hunting): dump: 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:31.328 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.2 v1:172.21.15.138:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91793735 0 0) 0x7f7250134b40 con 0x7f725006ab70 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f723c006d30 con 0x7f725006ab70 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.2 v1:172.21.15.138:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7248001f20 con 0x7f725006ab70 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient(hunting): dump: 2024-03-29T11:30:31.329 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1015406448 0 0) 0x7f72501db540 con 0x7f7250141fb0 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 >> v1:172.21.15.138:6790/0 conn(0x7f725006ab70 legacy=0x7f7250134430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 >> v1:172.21.15.152:6789/0 conn(0x7f7250070760 legacy=0x7f72501dae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 monclient: found mon.a 2024-03-29T11:30:31.330 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7250137180 con 0x7f7250141fb0 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.326+0000 7f72377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 10 monclient: _renew_subs 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7250137460 con 0x7f7250141fb0 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 10 monclient: _renew_subs 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:31.331 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7250136190 con 0x7f7250141fb0 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f725564e640 1 librados: init done 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72440047e0 con 0x7f7250141fb0 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f72377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.327+0000 7f724d7fa640 4 set_mon_vals no callback set 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.328+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7f7244001950 con 0x7f7250141fb0 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.328+0000 7f72377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.328+0000 7f72377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.328+0000 7f72377fe640 10 monclient: dump: 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:31.332 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.329+0000 7f725564e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.329+0000 7f725564e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.329+0000 7f725564e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.329+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7230005080 con 0x7f7250141fb0 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.329+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7f724401a7c0 con 0x7f7250141fb0 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.330+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7f7244084350 con 0x7f7250141fb0 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.330+0000 7f72377fe640 10 client.4587.objecter ms_dispatch 0x7f7250000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.330+0000 7f72377fe640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.330+0000 7f72377fe640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.330+0000 7f72377fe640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:30:31.333 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.331+0000 7f724effd640 10 monclient: get_auth_request con 0x7f723c06e7f0 auth_method 0 2024-03-29T11:30:31.334 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.331+0000 7f72377fe640 10 client.4587.objecter ms_handle_connect 0x7f723c06e7f0 2024-03-29T11:30:31.334 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.338+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f724404fba0 con 0x7f7250141fb0 2024-03-29T11:30:31.334 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.338+0000 7f72377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.334 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.338+0000 7f72377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:31.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.707+0000 7f725564e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:31.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.707+0000 7f725564e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:31.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.707+0000 7f725564e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:31.701 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.707+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7230005330 con 0x7f7250141fb0 2024-03-29T11:30:31.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.709+0000 7f72377fe640 1 -- 172.21.15.138:0/2941660212 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7f724404fba0 con 0x7f7250141fb0 2024-03-29T11:30:31.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.709+0000 7f72377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:31.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.709+0000 7f72377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:31.702 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:31.703 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.712+0000 7f725564e640 10 librados: watch_flush enter 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 10 librados: watch_flush exit 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 >> v1:172.21.15.152:6816/3052585572 conn(0x7f723c06e7f0 legacy=0x7f723c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 10 monclient: shutdown 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 >> v1:172.21.15.138:6789/0 conn(0x7f7250141fb0 legacy=0x7f72501da720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 shutdown_connections 2024-03-29T11:30:31.707 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 >> 172.21.15.138:0/2941660212 conn(0x7f7250067470 msgr2=0x7f7250069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:31.708 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 shutdown_connections 2024-03-29T11:30:31.708 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 1 -- 172.21.15.138:0/2941660212 wait complete. 2024-03-29T11:30:31.708 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:31.713+0000 7f725564e640 1 librados: shutdown 2024-03-29T11:30:31.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:38.527 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:38.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.686+0000 7ff37bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:30:38.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.686+0000 7ff37bfff640 10 monclient: build_initial_monmap 2024-03-29T11:30:38.680 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.686+0000 7ff37bfff640 10 monclient: monmap: 2024-03-29T11:30:38.680 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:38.680 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:38.680 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:38.687927+0000 2024-03-29T11:30:38.681 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:38.687927+0000 2024-03-29T11:30:38.681 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:38.681 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:38.681 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:38.681 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:38.681 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:38.681 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:38.682 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.687+0000 7ff37bfff640 1 Processor -- start 2024-03-29T11:30:38.682 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.687+0000 7ff37bfff640 1 -- start start 2024-03-29T11:30:38.682 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.687+0000 7ff37bfff640 10 monclient: init 2024-03-29T11:30:38.682 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.688+0000 7ff37bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:38.682 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.688+0000 7ff37bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 10 monclient(hunting): picked mon.noname-c con 0x7ff37c0aac30 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 10 monclient(hunting): picked mon.noname-a con 0x7ff37c072f20 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 10 monclient(hunting): picked mon.noname-b con 0x7ff37c06d7e0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c0ac6a0 con 0x7ff37c072f20 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c0ab230 con 0x7ff37c0aac30 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c0a9e90 con 0x7ff37c06d7e0 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.689+0000 7ff37bfff640 10 monclient(hunting): authenticate will time out at 1459.998673s 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37affd640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7ff37c0aac30 0x7ff37c13af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.138:59815/0 (socket says 172.21.15.138:59815) 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37affd640 1 -- 172.21.15.138:0/3673706248 learned_addr learned my addr 172.21.15.138:0/3673706248 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:38.688 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707839110 0 0) 0x7ff37c0ab230 con 0x7ff37c0aac30 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff368003520 con 0x7ff37c0aac30 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348880245 0 0) 0x7ff37c0ac6a0 con 0x7ff37c072f20 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.692+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c0ab230 con 0x7ff37c072f20 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169612700 0 0) 0x7ff368003520 con 0x7ff37c0aac30 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 >> v1:172.21.15.138:6790/0 conn(0x7ff37c06d7e0 legacy=0x7ff37c149750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:38.689 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 >> v1:172.21.15.138:6789/0 conn(0x7ff37c072f20 legacy=0x7ff37c0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.152:6789/0 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c0aa020 con 0x7ff37c0aac30 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37bfff640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff36c001690 con 0x7ff37c0aac30 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:38.690 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37a7fc640 10 monclient: dump: 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.693+0000 7ff37bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff36c0017f0 con 0x7ff37c0aac30 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:38.691 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37bfff640 4 set_mon_vals no callback set 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37a7fc640 1 -- 172.21.15.138:0/3673706248 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff36c001c40 con 0x7ff37c0aac30 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.694+0000 7ff37a7fc640 10 monclient: dump: 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:38.692 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:38.693 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.695+0000 7ff37bfff640 10 monclient: shutdown 2024-03-29T11:30:38.693 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.695+0000 7ff37bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:38.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.695+0000 7ff37bfff640 1 -- 172.21.15.138:0/3673706248 >> v1:172.21.15.152:6789/0 conn(0x7ff37c0aac30 legacy=0x7ff37c13af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.698+0000 7ff37bfff640 1 -- 172.21.15.138:0/3673706248 shutdown_connections 2024-03-29T11:30:38.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.698+0000 7ff37bfff640 1 -- 172.21.15.138:0/3673706248 >> 172.21.15.138:0/3673706248 conn(0x7ff37c0b3a30 msgr2=0x7ff37c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:38.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.699+0000 7ff37bfff640 1 -- 172.21.15.138:0/3673706248 shutdown_connections 2024-03-29T11:30:38.694 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.699+0000 7ff37bfff640 1 -- 172.21.15.138:0/3673706248 wait complete. 2024-03-29T11:30:38.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.704+0000 7ff37bfff640 10 monclient: build_initial_monmap 2024-03-29T11:30:38.698 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.704+0000 7ff37bfff640 10 monclient: monmap: 2024-03-29T11:30:38.698 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:38.698 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.705+0000 7ff37bfff640 1 librados: starting msgr at 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.705+0000 7ff37bfff640 1 librados: starting objecter 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.705+0000 7ff37bfff640 1 Processor -- start 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.705+0000 7ff37bfff640 1 -- start start 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.705+0000 7ff37bfff640 1 librados: setting wanted keys 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.705+0000 7ff37bfff640 1 librados: calling monclient init 2024-03-29T11:30:38.699 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.705+0000 7ff37bfff640 10 monclient: init 2024-03-29T11:30:38.700 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.706+0000 7ff37bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:38.700 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.706+0000 7ff37bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:38.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.708+0000 7ff37bfff640 10 monclient(hunting): picked mon.noname-a con 0x7ff37c06d7e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:38.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.708+0000 7ff37affd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ff37c06d7e0 0x7ff37c13e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:39191/0 (socket says 172.21.15.138:39191) 2024-03-29T11:30:38.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.708+0000 7ff37affd640 1 -- 172.21.15.138:0/174836868 learned_addr learned my addr 172.21.15.138:0/174836868 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:38.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.708+0000 7ff3737fe640 10 client.?.objecter ms_handle_connect 0x7ff37c06d7e0 2024-03-29T11:30:38.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.708+0000 7ff3737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:38.702 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.709+0000 7ff37bfff640 10 monclient(hunting): picked mon.noname-c con 0x7ff37c072f20 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.711+0000 7ff37bfff640 10 monclient(hunting): picked mon.noname-b con 0x7ff37c0aac30 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.711+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c13d950 con 0x7ff37c06d7e0 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.711+0000 7ff3737fe640 10 client.?.objecter ms_handle_connect 0x7ff37c0aac30 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.711+0000 7ff3737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.711+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c13c400 con 0x7ff37c0aac30 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289464444 0 0) 0x7ff37c13d950 con 0x7ff37c06d7e0 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c1daf90 con 0x7ff37c072f20 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff37bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff37bfff640 10 monclient(hunting): authenticate will time out at 1460.021377s 2024-03-29T11:30:38.710 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff35c004eb0 con 0x7ff37c06d7e0 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572209212 0 0) 0x7ff37c13c400 con 0x7ff37c0aac30 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c13d950 con 0x7ff37c0aac30 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.712+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45138691 0 0) 0x7ff35c004eb0 con 0x7ff37c06d7e0 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff37c13c400 con 0x7ff37c06d7e0 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff36c001bc0 con 0x7ff37c06d7e0 2024-03-29T11:30:38.711 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 10 monclient(hunting): dump: 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:38.712 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 393518105 0 0) 0x7ff37c13d950 con 0x7ff37c0aac30 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff35c004eb0 con 0x7ff37c0aac30 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff374001b30 con 0x7ff37c0aac30 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 10 monclient(hunting): dump: 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:38.713 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1053520981 0 0) 0x7ff37c13c400 con 0x7ff37c06d7e0 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.713+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 >> v1:172.21.15.138:6790/0 conn(0x7ff37c072f20 legacy=0x7ff37c1db480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 >> v1:172.21.15.152:6789/0 conn(0x7ff37c0aac30 legacy=0x7ff37c1e67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 1 monclient: found mon.a 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c13c090 con 0x7ff37c06d7e0 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff37bfff640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:30:38.714 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff37bfff640 10 monclient: _renew_subs 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff36c004a90 con 0x7ff37c06d7e0 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff37bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c13a8f0 con 0x7ff37c06d7e0 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff36c004da0 con 0x7ff37c06d7e0 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.714+0000 7ff3737fe640 10 monclient: dump: 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.715 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.715+0000 7ff37bfff640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.715+0000 7ff37bfff640 10 monclient: _renew_subs 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.715+0000 7ff37bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.715+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37c13be50 con 0x7ff37c06d7e0 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.715+0000 7ff37bfff640 1 librados: init done 2024-03-29T11:30:38.716 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.716+0000 7ff379ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.717+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff36c01ad70 con 0x7ff37c06d7e0 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.717+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7ff36c084a90 con 0x7ff37c06d7e0 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.717+0000 7ff3737fe640 10 client.4593.objecter ms_dispatch 0x7ff37c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.717+0000 7ff3737fe640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.717+0000 7ff3737fe640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.717+0000 7ff3737fe640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.718+0000 7ff37bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.718+0000 7ff37bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:38.718 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.718+0000 7ff37bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:38.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.718+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff360005080 con 0x7ff37c06d7e0 2024-03-29T11:30:38.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.718+0000 7ff373fff640 10 monclient: get_auth_request con 0x7ff35c06de80 auth_method 0 2024-03-29T11:30:38.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.719+0000 7ff3737fe640 10 client.4593.objecter ms_handle_connect 0x7ff35c06de80 2024-03-29T11:30:38.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.726+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff36c0501a0 con 0x7ff37c06d7e0 2024-03-29T11:30:38.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.726+0000 7ff3737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:38.719 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:38.726+0000 7ff3737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:39.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.069+0000 7ff37bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:39.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.069+0000 7ff37bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:39.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.069+0000 7ff37bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:39.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.069+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff360005330 con 0x7ff37c06d7e0 2024-03-29T11:30:39.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.070+0000 7ff3737fe640 1 -- 172.21.15.138:0/174836868 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7ff36c001210 con 0x7ff37c06d7e0 2024-03-29T11:30:39.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.070+0000 7ff3737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:39.063 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.070+0000 7ff3737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:39.063 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:39.063 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 10 librados: watch_flush enter 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 10 librados: watch_flush exit 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 10 client.4593.objecter successfully canceled tick 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff35c06de80 legacy=0x7ff35c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 10 monclient: shutdown 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 >> v1:172.21.15.138:6789/0 conn(0x7ff37c06d7e0 legacy=0x7ff37c13e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 shutdown_connections 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.073+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 >> 172.21.15.138:0/174836868 conn(0x7ff37c0b3a30 msgr2=0x7ff37c072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:39.067 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.074+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 shutdown_connections 2024-03-29T11:30:39.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.074+0000 7ff37bfff640 1 -- 172.21.15.138:0/174836868 wait complete. 2024-03-29T11:30:39.068 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:39.074+0000 7ff37bfff640 1 librados: shutdown 2024-03-29T11:30:39.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:45.890 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.042+0000 7fce5f98f640 10 monclient: get_monmap_and_config 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.042+0000 7fce5f98f640 10 monclient: build_initial_monmap 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.042+0000 7fce5f98f640 10 monclient: monmap: 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:46.042882+0000 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:46.042882+0000 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:46.036 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:46.037 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:46.037 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:46.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.043+0000 7fce5f98f640 1 Processor -- start 2024-03-29T11:30:46.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.043+0000 7fce5f98f640 1 -- start start 2024-03-29T11:30:46.037 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.043+0000 7fce5f98f640 10 monclient: init 2024-03-29T11:30:46.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.044+0000 7fce5f98f640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:46.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.044+0000 7fce5f98f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:46.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.044+0000 7fce5f98f640 10 monclient(hunting): picked mon.noname-b con 0x7fce58071de0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:46.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.044+0000 7fce5f98f640 10 monclient(hunting): picked mon.noname-c con 0x7fce580721c0 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:46.039 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.044+0000 7fce5f98f640 10 monclient(hunting): picked mon.noname-a con 0x7fce58145ae0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce5f98f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce58145650 con 0x7fce58145ae0 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce5f98f640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580ae260 con 0x7fce580721c0 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce5f98f640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce581420d0 con 0x7fce58071de0 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce5f98f640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce5f98f640 10 monclient(hunting): authenticate will time out at 1467.353094s 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce5df2c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fce58145ae0 0x7fce58145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:52506/0 (socket says 172.21.15.138:52506) 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce5df2c640 1 -- 172.21.15.138:0/3293286947 learned_addr learned my addr 172.21.15.138:0/3293286947 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.045+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749715727 0 0) 0x7fce58145650 con 0x7fce58145ae0 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce44003520 con 0x7fce58145ae0 2024-03-29T11:30:46.040 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3587382144 0 0) 0x7fce44003520 con 0x7fce58145ae0 2024-03-29T11:30:46.041 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 >> v1:172.21.15.138:6790/0 conn(0x7fce58071de0 legacy=0x7fce580aed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:46.041 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.046+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 >> v1:172.21.15.152:6789/0 conn(0x7fce580721c0 legacy=0x7fce5813f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:46.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:46.043 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 1 monclient: found mon.noname-a 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580b0590 con 0x7fce58145ae0 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fce48001670 con 0x7fce58145ae0 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.048+0000 7fce4ffff640 10 monclient: dump: 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:46.044 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.049+0000 7fce5f98f640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.049+0000 7fce5f98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.049+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce480018d0 con 0x7fce58145ae0 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.049+0000 7fce4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.049+0000 7fce5f98f640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.049+0000 7fce5f98f640 4 set_mon_vals no callback set 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.050+0000 7fce5f98f640 10 monclient: shutdown 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.050+0000 7fce5f98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:46.045 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.050+0000 7fce5f98f640 1 -- 172.21.15.138:0/3293286947 >> v1:172.21.15.138:6789/0 conn(0x7fce58145ae0 legacy=0x7fce58145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.050+0000 7fce4ffff640 1 -- 172.21.15.138:0/3293286947 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fce48001ca0 con 0x7fce58145ae0 2024-03-29T11:30:46.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.050+0000 7fce4ffff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:30:46.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.051+0000 7fce5f98f640 1 -- 172.21.15.138:0/3293286947 shutdown_connections 2024-03-29T11:30:46.046 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.051+0000 7fce5f98f640 1 -- 172.21.15.138:0/3293286947 >> 172.21.15.138:0/3293286947 conn(0x7fce5806b660 msgr2=0x7fce5806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.051+0000 7fce5f98f640 1 -- 172.21.15.138:0/3293286947 shutdown_connections 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.051+0000 7fce5f98f640 1 -- 172.21.15.138:0/3293286947 wait complete. 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.052+0000 7fce5f98f640 10 monclient: build_initial_monmap 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.052+0000 7fce5f98f640 10 monclient: monmap: 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:46.047 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.053+0000 7fce5f98f640 1 librados: starting msgr at 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.053+0000 7fce5f98f640 1 librados: starting objecter 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.053+0000 7fce5f98f640 1 Processor -- start 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.053+0000 7fce5f98f640 1 -- start start 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.053+0000 7fce5f98f640 1 librados: setting wanted keys 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.053+0000 7fce5f98f640 1 librados: calling monclient init 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.053+0000 7fce5f98f640 10 monclient: init 2024-03-29T11:30:46.048 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 10 monclient(hunting): picked mon.noname-c con 0x7fce58071de0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 10 monclient(hunting): picked mon.noname-a con 0x7fce58145ae0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 10 monclient(hunting): picked mon.noname-b con 0x7fce58137c60 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce581380d0 con 0x7fce58145ae0 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce581392b0 con 0x7fce58137c60 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce581d4cc0 con 0x7fce58071de0 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5f98f640 10 monclient(hunting): authenticate will time out at 1467.362390s 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5cf2a640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fce58145ae0 0x7fce58137550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33808/0 (socket says 172.21.15.138:33808) 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce5cf2a640 1 -- 172.21.15.138:0/344553545 learned_addr learned my addr 172.21.15.138:0/344553545 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce4dffb640 10 client.?.objecter ms_handle_connect 0x7fce58145ae0 2024-03-29T11:30:46.049 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.054+0000 7fce4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.056+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277242826 0 0) 0x7fce54001430 con 0x7fce58145ae0 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.056+0000 7fce4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.056+0000 7fce4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.056+0000 7fce4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.056+0000 7fce4dffb640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.056+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3c004eb0 con 0x7fce58145ae0 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 client.?.objecter ms_handle_connect 0x7fce58071de0 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1638432259 0 0) 0x7fce3c004eb0 con 0x7fce58145ae0 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce54001430 con 0x7fce58145ae0 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fce54001be0 con 0x7fce58145ae0 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.052 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): dump: 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709113825 0 0) 0x7fce581d4cc0 con 0x7fce58071de0 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:46.053 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.057+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3c004eb0 con 0x7fce58071de0 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4094180245 0 0) 0x7fce54001430 con 0x7fce58145ae0 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 >> v1:172.21.15.138:6790/0 conn(0x7fce58071de0 legacy=0x7fce581d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 >> v1:172.21.15.152:6789/0 conn(0x7fce58137c60 legacy=0x7fce581e1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 1 monclient: found mon.a 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce581d7f10 con 0x7fce58145ae0 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:46.054 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:46.055 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce5f98f640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce5f98f640 10 monclient: _renew_subs 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce5f98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.058+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce581e3090 con 0x7fce58145ae0 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce5f98f640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce5f98f640 10 monclient: _renew_subs 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce5f98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce581e3400 con 0x7fce58145ae0 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce5f98f640 1 librados: init done 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce54002170 con 0x7fce58145ae0 2024-03-29T11:30:46.057 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7fce54004e20 con 0x7fce58145ae0 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce4dffb640 10 monclient: dump: 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:46.058 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.059+0000 7fce4f7fe640 4 set_mon_vals no callback set 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.061+0000 7fce5f98f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.061+0000 7fce5f98f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.061+0000 7fce5f98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.061+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce38005080 con 0x7fce58145ae0 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.062+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7fce5401adf0 con 0x7fce58145ae0 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.066+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7fce54084dd0 con 0x7fce58145ae0 2024-03-29T11:30:46.059 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.066+0000 7fce4dffb640 10 client.4599.objecter ms_dispatch 0x7fce58000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:46.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.066+0000 7fce4dffb640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:46.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.066+0000 7fce4dffb640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:46.060 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.066+0000 7fce4dffb640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T11:30:46.061 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.068+0000 7fce5d72b640 10 monclient: get_auth_request con 0x7fce3c06dd60 auth_method 0 2024-03-29T11:30:46.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.069+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce540504e0 con 0x7fce58145ae0 2024-03-29T11:30:46.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.070+0000 7fce4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:46.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.070+0000 7fce4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:46.062 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.070+0000 7fce4dffb640 10 client.4599.objecter ms_handle_connect 0x7fce3c06dd60 2024-03-29T11:30:46.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.392+0000 7fce5f98f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:46.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.392+0000 7fce5f98f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:46.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.392+0000 7fce5f98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:46.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.392+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce38005330 con 0x7fce58145ae0 2024-03-29T11:30:46.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.397+0000 7fce4dffb640 1 -- 172.21.15.138:0/344553545 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7fce540525e0 con 0x7fce58145ae0 2024-03-29T11:30:46.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.397+0000 7fce4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:46.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.397+0000 7fce4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:46.390 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:46.390 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:46.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.400+0000 7fce5f98f640 10 librados: watch_flush enter 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 10 librados: watch_flush exit 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 10 client.4599.objecter successfully canceled tick 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 >> v1:172.21.15.152:6816/3052585572 conn(0x7fce3c06dd60 legacy=0x7fce3c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 10 monclient: shutdown 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 >> v1:172.21.15.138:6789/0 conn(0x7fce58145ae0 legacy=0x7fce58137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 shutdown_connections 2024-03-29T11:30:46.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.401+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 >> 172.21.15.138:0/344553545 conn(0x7fce5806b660 msgr2=0x7fce580b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:46.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.402+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 shutdown_connections 2024-03-29T11:30:46.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.402+0000 7fce5f98f640 1 -- 172.21.15.138:0/344553545 wait complete. 2024-03-29T11:30:46.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:46.402+0000 7fce5f98f640 1 librados: shutdown 2024-03-29T11:30:46.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:53.225 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:53.371 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.377+0000 7ff5d86d9640 10 monclient: get_monmap_and_config 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.377+0000 7ff5d86d9640 10 monclient: build_initial_monmap 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.377+0000 7ff5d86d9640 10 monclient: monmap: 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:30:53.379217+0000 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:30:53.379217+0000 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-c 2024-03-29T11:30:53.372 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-b 2024-03-29T11:30:53.373 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:53.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.378+0000 7ff5d86d9640 1 Processor -- start 2024-03-29T11:30:53.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.379+0000 7ff5d86d9640 1 -- start start 2024-03-29T11:30:53.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.379+0000 7ff5d86d9640 10 monclient: init 2024-03-29T11:30:53.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.380+0000 7ff5d86d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:53.374 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.380+0000 7ff5d86d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:53.376 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.380+0000 7ff5d86d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff5c80389e0 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.383+0000 7ff5d86d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff5c8038dc0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d6475640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ff5c80389e0 0x7ff5c8033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:40375/0 (socket says 172.21.15.138:40375) 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d6475640 1 -- 172.21.15.138:0/335963841 learned_addr learned my addr 172.21.15.138:0/335963841 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d86d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff5c8030660 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80c54c0 con 0x7ff5c80389e0 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c8036430 con 0x7ff5c8030660 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c8039840 con 0x7ff5c8038dc0 2024-03-29T11:30:53.380 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d86d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d86d9640 10 monclient(hunting): authenticate will time out at 1474.693546s 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527120889 0 0) 0x7ff5c80c54c0 con 0x7ff5c80389e0 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d5473640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d5473640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d5473640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d5473640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.384+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5bc003520 con 0x7ff5c80389e0 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.385+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 <== mon.2 v1:172.21.15.138:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328681089 0 0) 0x7ff5c8039840 con 0x7ff5c8038dc0 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.385+0000 7ff5d5473640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.385+0000 7ff5d5473640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.385+0000 7ff5d5473640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.385+0000 7ff5d5473640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.385+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 --> v1:172.21.15.138:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c80c54c0 con 0x7ff5c8038dc0 2024-03-29T11:30:53.381 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.386+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4205529966 0 0) 0x7ff5bc003520 con 0x7ff5c80389e0 2024-03-29T11:30:53.382 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.386+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 >> v1:172.21.15.138:6790/0 conn(0x7ff5c8038dc0 legacy=0x7ff5c80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.382 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.386+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 >> v1:172.21.15.152:6789/0 conn(0x7ff5c8030660 legacy=0x7ff5c80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.382 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.386+0000 7ff5d5473640 10 monclient: _finish_hunting 0 2024-03-29T11:30:53.382 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.386+0000 7ff5d5473640 1 monclient: found mon.noname-a 2024-03-29T11:30:53.383 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.386+0000 7ff5d5473640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:53.383 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.386+0000 7ff5d5473640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.138:6789/0 2024-03-29T11:30:53.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c8035690 con 0x7ff5c80389e0 2024-03-29T11:30:53.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 10 monclient: _finish_auth 0 2024-03-29T11:30:53.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 10 monclient: _check_auth_tickets 2024-03-29T11:30:53.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:53.385 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff5d004f730 con 0x7ff5c80389e0 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d5473640 10 monclient: dump: 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.386 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d86d9640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.387+0000 7ff5d86d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 <== mon.0 v1:172.21.15.138:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5d0075f10 con 0x7ff5c80389e0 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d5473640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d5473640 1 -- 172.21.15.138:0/335963841 <== mon.0 v1:172.21.15.138:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff5d00758b0 con 0x7ff5c80389e0 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d86d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:53.387 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d86d9640 4 set_mon_vals no callback set 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d5473640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d5473640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.389+0000 7ff5d5473640 10 monclient: dump: 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:53.388 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:53.389 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:53.389 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:53.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.390+0000 7ff5d86d9640 10 monclient: shutdown 2024-03-29T11:30:53.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.390+0000 7ff5d86d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:53.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.390+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 >> v1:172.21.15.138:6789/0 conn(0x7ff5c80389e0 legacy=0x7ff5c8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.390+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 shutdown_connections 2024-03-29T11:30:53.389 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.390+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 >> 172.21.15.138:0/335963841 conn(0x7ff5c8005c00 msgr2=0x7ff5c8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:53.390 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.393+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 shutdown_connections 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.393+0000 7ff5d86d9640 1 -- 172.21.15.138:0/335963841 wait complete. 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 10 monclient: build_initial_monmap 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 10 monclient: monmap: 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a 2024-03-29T11:30:53.391 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.noname-b 2024-03-29T11:30:53.392 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.noname-c 2024-03-29T11:30:53.392 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:53.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 1 librados: starting msgr at 2024-03-29T11:30:53.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 1 librados: starting objecter 2024-03-29T11:30:53.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 1 Processor -- start 2024-03-29T11:30:53.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 1 -- start start 2024-03-29T11:30:53.392 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 1 librados: setting wanted keys 2024-03-29T11:30:53.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 1 librados: calling monclient init 2024-03-29T11:30:53.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.394+0000 7ff5d86d9640 10 monclient: init 2024-03-29T11:30:53.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.395+0000 7ff5d86d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:53.393 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.395+0000 7ff5d86d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:53.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff5c8030660 addr v1:172.21.15.138:6789/0 2024-03-29T11:30:53.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff5c80389e0 addr v1:172.21.15.138:6790/0 2024-03-29T11:30:53.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff5c80c5b30 addr v1:172.21.15.152:6789/0 2024-03-29T11:30:53.394 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80c7170 con 0x7ff5c8030660 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80c6160 con 0x7ff5c80c5b30 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 1 -- --> v1:172.21.15.138:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80c73e0 con 0x7ff5c80389e0 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d86d9640 10 monclient(hunting): authenticate will time out at 1474.707637s 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d6475640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ff5c8030660 0x7ff5c802f9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:40843/0 (socket says 172.21.15.138:40843) 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5d6475640 1 -- 172.21.15.138:0/3894408489 learned_addr learned my addr 172.21.15.138:0/3894408489 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5c6ffd640 10 client.?.objecter ms_handle_connect 0x7ff5c8030660 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.398+0000 7ff5c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677898768 0 0) 0x7ff5c80c7170 con 0x7ff5c8030660 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.395 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b4004eb0 con 0x7ff5c8030660 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985710757 0 0) 0x7ff5b4004eb0 con 0x7ff5c8030660 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5c80c7170 con 0x7ff5c8030660 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff5d004f400 con 0x7ff5c8030660 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.396 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.399+0000 7ff5c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.400+0000 7ff5c6ffd640 10 client.?.objecter ms_handle_connect 0x7ff5c80c5b30 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.400+0000 7ff5c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.400+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512360558 0 0) 0x7ff5c80c7170 con 0x7ff5c8030660 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.400+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 >> v1:172.21.15.138:6790/0 conn(0x7ff5c80389e0 legacy=0x7ff5c81603b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:53.397 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 >> v1:172.21.15.152:6789/0 conn(0x7ff5c80c5b30 legacy=0x7ff5c81675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 1 monclient: found mon.a 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c8160ac0 con 0x7ff5c8030660 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 5 monclient: authenticate success, global_id 4605 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 10 monclient: _renew_subs 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c8160d10 con 0x7ff5c8030660 2024-03-29T11:30:53.398 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:53.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 10 monclient: _renew_subs 2024-03-29T11:30:53.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:53.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.401+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c8168ee0 con 0x7ff5c8030660 2024-03-29T11:30:53.399 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.402+0000 7ff5d86d9640 1 librados: init done 2024-03-29T11:30:53.400 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5d004fdd0 con 0x7ff5c8030660 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 971009722 0 0) 0x7ff5d0067bf0 con 0x7ff5c8030660 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5c6ffd640 10 monclient: dump: 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:fsid 113c9c6c-c44b-4755-b507-344fd243c0cb 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:created 2024-03-29T11:19:17.751191+0000 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.a 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.152:6789/0 mon.b 2024-03-29T11:30:53.401 INFO:teuthology.misc.health.smithi138.stderr:2: v1:172.21.15.138:6790/0 mon.c 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr: 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5d4c72640 4 set_mon_vals no callback set 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.404+0000 7ff5d86d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.405+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4240087502 0 0) 0x7ff5d0067e10 con 0x7ff5c8030660 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2986286676 0 0) 0x7ff5d00ff6e0 con 0x7ff5c8030660 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5c6ffd640 10 client.4605.objecter ms_dispatch 0x7ff5d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5c6ffd640 3 client.4605.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5c6ffd640 3 client.4605.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5c6ffd640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.405+0000 7ff5d86d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5d86d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:53.402 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b8005080 con 0x7ff5c8030660 2024-03-29T11:30:53.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.406+0000 7ff5d5c74640 10 monclient: get_auth_request con 0x7ff5b406da10 auth_method 0 2024-03-29T11:30:53.403 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.407+0000 7ff5c6ffd640 10 client.4605.objecter ms_handle_connect 0x7ff5b406da10 2024-03-29T11:30:53.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.415+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5d00cafc0 con 0x7ff5c8030660 2024-03-29T11:30:53.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.415+0000 7ff5c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:53.409 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.415+0000 7ff5c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:53.735 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.742+0000 7ff5d86d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:53.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.742+0000 7ff5d86d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:53.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.742+0000 7ff5d86d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.138:6789/0 2024-03-29T11:30:53.736 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.742+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b8005330 con 0x7ff5c8030660 2024-03-29T11:30:53.738 INFO:teuthology.misc.health.smithi138.stdout: 2024-03-29T11:30:53.738 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":11,"fsid":"113c9c6c-c44b-4755-b507-344fd243c0cb","created":"2024-03-29T11:19:55.913327+0000","modified":"2024-03-29T11:30:00.837863+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:00.837863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d49822d-afc6-4c38-9d56-52b09808838f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4ec3ef4f-0970-4f8e-ad05-505a67a88dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"281d93c3-d8e5-4bf5-afeb-805e53e9d2be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4defe436-16df-4982-aedd-de68b04aeb07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"02d2a411-15db-4cda-9b3d-9d9176a7c4d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef6a6f95-11b2-4bcf-85aa-8bff1ca8c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"43d86222-0bbc-4ba2-aee1-5c47d9f26662","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"05f0b6ff-da6f-45dc-8820-500924b513a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:53.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.744+0000 7ff5c6ffd640 1 -- 172.21.15.138:0/3894408489 <== mon.0 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208155230) 0x7ff5d00ccef0 con 0x7ff5c8030660 2024-03-29T11:30:53.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.744+0000 7ff5c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:53.738 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.744+0000 7ff5c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:53.741 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.747+0000 7ff5d86d9640 10 librados: watch_flush enter 2024-03-29T11:30:53.742 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.747+0000 7ff5d86d9640 10 librados: watch_flush exit 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.747+0000 7ff5d86d9640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.747+0000 7ff5d86d9640 10 client.4605.objecter successfully canceled tick 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.748+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 >> v1:172.21.15.152:6816/3052585572 conn(0x7ff5b406da10 legacy=0x7ff5b406fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.748+0000 7ff5d86d9640 10 monclient: shutdown 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.748+0000 7ff5d86d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.748+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 >> v1:172.21.15.138:6789/0 conn(0x7ff5c8030660 legacy=0x7ff5c802f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.748+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 shutdown_connections 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.748+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 >> 172.21.15.138:0/3894408489 conn(0x7ff5c8005c00 msgr2=0x7ff5c8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.749+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 shutdown_connections 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.749+0000 7ff5d86d9640 1 -- 172.21.15.138:0/3894408489 wait complete. 2024-03-29T11:30:53.743 INFO:teuthology.misc.health.smithi138.stderr:2024-03-29T11:30:53.749+0000 7ff5d86d9640 1 librados: shutdown 2024-03-29T11:30:53.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:53.759 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:53.761 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:53.761 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:53.761 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:53.792 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:53.792 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:53.792 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:53.864 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:53.864 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:53.864 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:53.933 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:53.934 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:53.934 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.011 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:54.011 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:54.011 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.053 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:54.054 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:54.054 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.130 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:54.130 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:54.130 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.205 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:54.205 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:54.206 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.274 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:54.274 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:54.274 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:54.274 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.298 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:54.298 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:54.298 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.338 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:54.338 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:54.338 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:54.339 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.350 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:54.350 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:54.350 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.402 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:54.402 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:54.402 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:54.413 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:54.413 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:54.413 DEBUG:teuthology.orchestra.run.smithi138:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:54.475 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:30:54.476 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:00.163 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:31:00.163 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:00.275 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:31:00.275 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:00.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:31:00.334 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:00.409 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:31:00.409 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:04.347 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:31:04.347 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:04.451 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:31:04.451 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:04.507 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:31:04.507 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:04.574 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:31:04.575 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/data/mon.a.tgz 2024-03-29T11:31:04.575 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-03-29T11:31:04.594 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.gEaZx9V11c 2024-03-29T11:31:04.594 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.gEaZx9V11c 2024-03-29T11:31:04.790 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.gEaZx9V11c 2024-03-29T11:31:04.879 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.gEaZx9V11c is 126KB 2024-03-29T11:31:04.904 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.gEaZx9V11c 2024-03-29T11:31:04.919 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/data/mon.c.tgz 2024-03-29T11:31:04.919 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-03-29T11:31:04.974 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.m6XeHnm3lu 2024-03-29T11:31:04.974 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.m6XeHnm3lu 2024-03-29T11:31:05.061 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.m6XeHnm3lu 2024-03-29T11:31:05.145 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.m6XeHnm3lu is 126KB 2024-03-29T11:31:05.166 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.m6XeHnm3lu 2024-03-29T11:31:05.180 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/data/mon.b.tgz 2024-03-29T11:31:05.181 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-03-29T11:31:05.198 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.gpBZEDO4eZ 2024-03-29T11:31:05.199 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.gpBZEDO4eZ 2024-03-29T11:31:05.303 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.gpBZEDO4eZ 2024-03-29T11:31:05.391 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.gpBZEDO4eZ is 126KB 2024-03-29T11:31:05.414 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.gpBZEDO4eZ 2024-03-29T11:31:05.429 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:31:05.430 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:05.433 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:05.512 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:05.551 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:05.551 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:31:05.553 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/remote/smithi138/crash 2024-03-29T11:31:05.554 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:05.581 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/remote/smithi152/crash 2024-03-29T11:31:05.582 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:05.617 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:31:05.617 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:05.625 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:05.687 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:31:05.687 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39925.log 2024-03-29T11:31:05.687 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38911.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.39925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41231.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39925.log.gz 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41231.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38911.log.gz 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40801.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39496.log: gzip -5 --verbose 86.7% -- /var/log/ceph/ceph-client.admin.39301.log -- replaced with /var/log/ceph/ceph-client.admin.41231.log.gz 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34628.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40801.log.gz 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.34628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39535.log 2024-03-29T11:31:05.688 INFO:teuthology.orchestra.run.smithi138.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40449.log 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34628.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40996.log 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39691.log 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39535.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38167.log 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40449.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40996.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38167.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39691.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39769.log 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41192.log 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38167.log.gz 2024-03-29T11:31:05.689 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40254.log 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41192.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39769.log.gz 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40722.log 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38089.log 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41192.log.gz 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39652.log 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40254.log.gz 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40163.log 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40722.log.gz 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38365.log 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38089.log.gz 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39652.log.gz 2024-03-29T11:31:05.690 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38365.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40163.log.gz 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38794.log 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38365.log.gz 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38794.log.gz 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38638.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38677.log 85.8% 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz -5 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39145.log: gzip/var/log/ceph/ceph-client.admin.38677.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39886.log 2024-03-29T11:31:05.691 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37933.log 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38677.log.gz 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37933.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39886.log.gz 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40840.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37933.log.gz 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39808.log 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38248.log 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39964.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39808.log.gz 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39067.log 2024-03-29T11:31:05.692 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37972.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38248.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38248.log.gz 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37972.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39067.log.gz 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40120.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40003.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.37972.log.gz 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38287.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40120.log: gzip -5 --verbose -- 86.7% /var/log/ceph/ceph-client.admin.38443.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38287.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-03-29T11:31:05.693 INFO:teuthology.orchestra.run.smithi138.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38287.log.gz 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34337.log 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38443.log.gz 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39223.log 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40644.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.34337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39223.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34337.log.gz 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose 85.7% -- /var/log/ceph/ceph-client.admin.41113.log -- replaced with /var/log/ceph/ceph-client.admin.39223.log.gz 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38989.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2024-03-29T11:31:05.694 INFO:teuthology.orchestra.run.smithi138.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41035.log 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39028.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41113.log.gz 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39106.log 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38011.log 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41035.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38716.log 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41035.log.gz 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38011.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39106.log.gz 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41270.log 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38716.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38011.log.gz 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41270.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38716.log.gz 2024-03-29T11:31:05.695 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41270.log.gz 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-mgr.y.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40527.log 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38209.log 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40527.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40527.log.gz 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38560.log 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5/var/log/ceph/ceph-client.admin.38950.log: --verbose -- /var/log/ceph/ceph-client.admin.41387.log 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38209.log.gz 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38560.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gz 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr: -- /var/log/ceph/ceph-client.admin.40213.log 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38560.log.gz 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40605.log 2024-03-29T11:31:05.696 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40213.log: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41387.log.gz 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41348.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41074.log: -- 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40605.log.gz 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr: /var/log/ceph/ceph-client.admin.40566.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41348.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40566.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41348.log.gz 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40371.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.0.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40566.log.gz 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40683.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41309.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40683.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40371.log.gz 2024-03-29T11:31:05.697 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41309.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40683.log.gz 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39847.log 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39184.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41309.log.gz 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38599.log 86.3% 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34731.log 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39847.log.gz 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38599.log: /var/log/ceph/ceph-client.admin.34731.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38599.log.gz 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34731.log.gz 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39418.log 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38833.log 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39418.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:31:05.698 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38833.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39418.log.gz 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38833.log.gz 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41153.log 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39379.log 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38521.log 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39379.log: gzip -5 86.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41153.log.gz /var/log/ceph/ceph-client.admin.38050.log 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38404.log 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39379.log.gz 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38521.log: /var/log/ceph/ceph-client.admin.38050.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38521.log.gz 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39730.log 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28518.log 2024-03-29T11:31:05.699 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38404.log: /var/log/ceph/ceph-client.admin.39730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39457.log 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.tmp-client.admin.28518.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28518.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38404.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39730.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39457.log: /var/log/ceph/ceph-client.admin.40081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38872.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39457.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40293.log 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38482.log 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40957.log: gzip/var/log/ceph/ceph-client.admin.40293.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38326.log 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38482.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40293.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-03-29T11:31:05.700 INFO:teuthology.orchestra.run.smithi138.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38482.log.gz 2024-03-29T11:31:05.701 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-03-29T11:31:05.701 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38755.log 2024-03-29T11:31:05.701 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38326.log: /var/log/ceph/ceph-client.admin.40410.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:31:05.701 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38755.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38326.log.gz 2024-03-29T11:31:05.701 INFO:teuthology.orchestra.run.smithi138.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-03-29T11:31:05.701 INFO:teuthology.orchestra.run.smithi138.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38755.log.gz 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34458.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34178.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34298.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.34178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34218.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34458.log.gz 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.34298.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34178.log.gz 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34258.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.34218.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34298.log.gz 2024-03-29T11:31:05.707 INFO:teuthology.orchestra.run.smithi152.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34218.log.gz 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.34258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34338.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34258.log.gz 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.34338.log: -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.5.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34338.log.gz 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.34418.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34378.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.34378.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-mon.b.log: 2024-03-29T11:31:05.708 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34378.log.gz 2024-03-29T11:31:05.764 INFO:teuthology.orchestra.run.smithi152.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:31:17.354 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:31:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:31:17.696 INFO:teuthology.orchestra.run.smithi152.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:31:17.726 INFO:teuthology.orchestra.run.smithi152.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:31:21.635 INFO:teuthology.orchestra.run.smithi138.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:31:21.758 INFO:teuthology.orchestra.run.smithi152.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:31:23.100 INFO:teuthology.orchestra.run.smithi138.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:31:23.195 INFO:teuthology.orchestra.run.smithi138.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:31:53.662 INFO:teuthology.orchestra.run.smithi138.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:32:02.040 INFO:teuthology.orchestra.run.smithi138.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:32:02.042 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-29T11:32:02.042 INFO:teuthology.orchestra.run.smithi138.stderr:real 0m56.404s 2024-03-29T11:32:02.042 INFO:teuthology.orchestra.run.smithi138.stderr:user 2m40.861s 2024-03-29T11:32:02.042 INFO:teuthology.orchestra.run.smithi138.stderr:sys 0m5.860s 2024-03-29T11:32:14.369 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:32:14.371 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-29T11:32:14.371 INFO:teuthology.orchestra.run.smithi152.stderr:real 1m8.694s 2024-03-29T11:32:14.371 INFO:teuthology.orchestra.run.smithi152.stderr:user 1m56.015s 2024-03-29T11:32:14.371 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m4.294s 2024-03-29T11:32:14.372 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:32:14.372 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/remote/smithi138/log 2024-03-29T11:32:14.373 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:39.001 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/remote/smithi152/log 2024-03-29T11:32:39.002 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:56.890 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:57.091 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fba2fbafe6194aed9a356e5e4224b9d8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:57.095 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:57.103 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:57.110 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:57.111 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:57.124 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:57.206 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:57.215 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:57.215 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:57.218 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:57.232 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:57.235 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:57.238 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:57.238 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-03-29T11:32:57.238 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +473us[ +473us] +/- 92ms 2024-03-29T11:32:57.238 INFO:teuthology.orchestra.run.smithi138.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -109us[ -117us] +/- 55ms 2024-03-29T11:32:57.238 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -273us[ -273us] +/- 87ms 2024-03-29T11:32:57.239 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:57.240 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:57.240 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-29T11:32:57.240 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +467us[ +466us] +/- 92ms 2024-03-29T11:32:57.240 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -110us[ -112us] +/- 55ms 2024-03-29T11:32:57.241 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -270us[ -271us] +/- 87ms 2024-03-29T11:32:57.241 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:57.241 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:57.250 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:57.250 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:57.258 DEBUG:teuthology.orchestra.run.smithi138:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:57.348 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:57.419 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:57.451 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:57.483 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:57.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:57.503 INFO:teuthology.task.internal:Duration was 1268.451635 seconds 2024-03-29T11:32:57.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:57.512 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:57.512 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:57.515 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:57.556 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:57.568 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:57.889 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:57.890 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-03-29T11:32:57.890 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:57.949 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-03-29T11:32:57.949 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:57.981 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:57.981 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:57.992 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:58.051 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:58.051 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:58.061 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:58.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:58.199 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:58.199 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:58.226 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:58.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:58.263 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:58.269 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:58.292 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-03-29T11:32:58.319 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-03-29T11:32:58.340 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:58.368 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:58.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:58.403 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:58.404 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/remote/smithi138 2024-03-29T11:32:58.405 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:58.445 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696/remote/smithi152 2024-03-29T11:32:58.445 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:58.480 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:58.480 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:58.483 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:58.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:58.541 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:58.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:58.551 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:58.551 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:58.554 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:58.566 INFO:teuthology.orchestra.run.smithi138.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:58.584 INFO:teuthology.orchestra.run.smithi152.stdout: 266041 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:58.585 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:58.595 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:58.615 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:58.653 DEBUG:teuthology.nuke:shortname: smithi138 2024-03-29T11:32:58.653 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:58.677 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.977449', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGpptLcC57lUuS0F5ZaBYanQ/mD3T/64LfMnxx3oNg5dHdHQtLjUnvP8USSCCr61/p4NtkvfPKrD8ypNlloLkw='} 2024-03-29T11:32:58.681 DEBUG:teuthology.nuke:shortname: smithi152 2024-03-29T11:32:58.682 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:58.700 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.978867', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAxy1Of+RLhQGS1byxOnbjm9Sof/8KrE4Eb13V1+djlQp3zjp6NipL3pg8z+KxDbrJI5ZuZbWw2almQYuUn85mc='} 2024-03-29T11:32:58.729 INFO:teuthology.orchestra.console.smithi138:Power off 2024-03-29T11:32:58.729 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:58.746 INFO:teuthology.orchestra.console.smithi152:Power off 2024-03-29T11:32:58.747 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:58.757 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:58.757 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:58.773 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:58.773 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:58.783 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-29T11:32:58.796 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-29T11:33:02.785 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:02.796 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:02.896 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-29T11:33:02.909 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-29T11:33:06.899 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:06.910 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:07.010 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-03-29T11:33:07.021 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-29T11:33:07.111 INFO:teuthology.orchestra.console.smithi138:Power off completed 2024-03-29T11:33:11.023 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:11.133 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-03-29T11:33:11.234 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-03-29T11:33:11.338 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1268.451634645462 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fba2fbafe6194aed9a356e5e4224b9d8 status: fail success: false 2024-03-29T11:33:11.339 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:33:11.428 INFO:teuthology.run:FAIL